2024-04-25T18:18:47.545 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T18:18:47.545 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} --name leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi --archive /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605 --verbose --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/orig.config.yaml 2024-04-25T18:18:47.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T18:18:47.690 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605 branch: wip-lusov-quiescer description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7673605' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiescer ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiescer sha1: 7797745e6d8b3ac036504dc61491d368615024da owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4419 sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f sleep_before_teardown: 0 subset: 200/111111 suite: fs suite_branch: wip-lusov-quiescer suite_path: /home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph suite_sha1: 7797745e6d8b3ac036504dc61491d368615024da syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKfD/L/4b3vfTg7Wgk2ts1MsTZ5nXjE5KE7kqkNSEZTL8B94ftKeRrYyw+T2CK6tX/I4vqit23hjc5ULKcbslp4= smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGyn7n5ycz2BRHDdwHvCt7Z0wYrZku3XnTjAS1GQqPqU814qFVnA+4d7VzeKtfAKImH/g2nP9m2luVnIK6NuuY= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPhyDAFFEzUC2YHEudlcfE0QYUfX7l9do1WA4GPZdx6d65OSHWRqp+7jZpO37BzXfXZFtZ8GYlYqIfH3ikmSw98= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_17:43:55 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T18:18:47.690 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa; will attempt to use it 2024-04-25T18:18:47.691 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks 2024-04-25T18:18:47.691 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T18:18:47.692 INFO:teuthology.task.internal:Checking packages... 2024-04-25T18:18:47.713 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd69f13d827952bb07bb9b7973bca14f97d764d6f' 2024-04-25T18:18:47.713 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T18:18:47.713 INFO:teuthology.packaging:ref: None 2024-04-25T18:18:47.713 INFO:teuthology.packaging:tag: None 2024-04-25T18:18:47.713 INFO:teuthology.packaging:branch: wip-lusov-quiescer 2024-04-25T18:18:47.713 INFO:teuthology.packaging:sha1: d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:18:47.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiescer 2024-04-25T18:18:47.866 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3156.gd69f13d8 2024-04-25T18:18:47.867 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T18:18:47.874 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T18:18:47.874 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T18:18:47.890 INFO:teuthology.task.internal:Saving configuration 2024-04-25T18:18:47.910 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T18:18:47.916 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T18:18:47.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.580732', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKfD/L/4b3vfTg7Wgk2ts1MsTZ5nXjE5KE7kqkNSEZTL8B94ftKeRrYyw+T2CK6tX/I4vqit23hjc5ULKcbslp4='} 2024-04-25T18:18:47.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.581410', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGyn7n5ycz2BRHDdwHvCt7Z0wYrZku3XnTjAS1GQqPqU814qFVnA+4d7VzeKtfAKImH/g2nP9m2luVnIK6NuuY='} 2024-04-25T18:18:47.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.579791', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPhyDAFFEzUC2YHEudlcfE0QYUfX7l9do1WA4GPZdx6d65OSHWRqp+7jZpO37BzXfXZFtZ8GYlYqIfH3ikmSw98='} 2024-04-25T18:18:47.983 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T18:18:47.988 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T18:18:47.988 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T18:18:47.989 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T18:18:47.989 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T18:18:48.084 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efeedf9b0d0>, signals=[15]) 2024-04-25T18:18:48.084 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T18:18:48.090 INFO:teuthology.task.internal:Opening connections... 2024-04-25T18:18:48.091 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:18:48.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:18:48.167 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-25T18:18:48.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:18:48.239 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-04-25T18:18:48.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:18:48.311 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T18:18:48.318 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-04-25T18:18:48.336 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-04-25T18:18:48.336 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-04-25T18:18:48.394 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-04-25T18:18:48.394 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-04-25T18:18:48.394 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-04-25T18:18:48.394 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T18:18:48.395 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T18:18:48.396 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-04-25T18:18:48.418 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-25T18:18:48.436 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-25T18:18:48.436 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-25T18:18:48.493 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T18:18:48.494 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T18:18:48.494 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T18:18:48.494 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-25T18:18:48.518 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-04-25T18:18:48.535 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-04-25T18:18:48.535 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-04-25T18:18:48.592 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T18:18:48.593 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T18:18:48.593 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T18:18:48.593 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-04-25T18:18:48.618 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T18:18:48.628 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T18:18:48.634 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T18:18:48.634 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-04-25T18:18:48.637 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-25T18:18:48.639 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-04-25T18:18:48.656 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T18:18:48.662 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T18:18:48.662 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-04-25T18:18:48.695 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-25T18:18:48.697 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-04-25T18:18:48.725 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T18:18:48.805 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T18:18:48.820 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T18:18:48.821 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-25T18:18:48.821 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-25T18:18:48.821 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T18:18:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:18:48.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-25T18:18:49.091 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T18:18:49.092 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T18:18:49.092 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-04-25T18:18:49.110 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-437.el9.x86_64 2024-04-25T18:18:49.110 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T18:18:49.110 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T18:18:49.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum upgrade -y linux-firmware 2024-04-25T18:18:51.384 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T18:18:53.610 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T18:18:57.869 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-25T18:18:59.425 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-04-25T18:19:00.743 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T18:19:06.010 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-25T18:19:08.029 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-25T18:19:08.030 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-04-25T18:19:08.030 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-25T18:19:08.132 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T18:19:08.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T18:19:08.277 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-25T18:19:08.278 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-25T18:19:08.280 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi063.front.sepia.ceph.com to complete... 2024-04-25T18:19:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25 18:19:56 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T18:19:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:FINISHED --2024-04-25 18:19:56-- 2024-04-25T18:19:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:Total wall clock time: 49s 2024-04-25T18:19:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:Downloaded: 1 files, 341M in 48s (7.10 MB/s) 2024-04-25T18:19:56.927 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-25T18:19:56.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T18:19:59.121 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying... ######################################## 2024-04-25T18:19:59.237 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing... ######################################## 2024-04-25T18:19:59.351 INFO:teuthology.orchestra.run.smithi063.stdout:Updating / installing... 2024-04-25T18:20:07.656 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T18:20:46.082 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T18:20:46.083 DEBUG:teuthology.orchestra.run.smithi063:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T18:20:46.166 INFO:teuthology.orchestra.run.smithi063.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.166 INFO:teuthology.orchestra.run.smithi063.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.166 INFO:teuthology.orchestra.run.smithi063.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.166 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T18:20:46.166 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T18:20:46.167 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T18:20:46.168 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T18:20:46.169 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T18:20:46.170 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T18:20:46.171 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T18:20:46.172 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T18:20:46.173 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T18:20:46.174 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T18:20:46.175 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T18:20:46.176 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T18:20:46.177 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T18:20:46.178 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T18:20:46.179 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T18:20:46.180 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T18:20:46.181 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T18:20:46.182 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T18:20:46.183 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T18:20:46.184 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T18:20:46.185 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T18:20:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T18:20:46.187 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T18:20:46.188 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T18:20:46.189 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T18:20:46.190 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T18:20:46.191 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T18:20:46.192 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T18:20:46.193 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T18:20:46.194 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T18:20:46.195 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T18:20:46.196 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T18:20:46.197 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T18:20:46.198 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T18:20:46.199 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T18:20:46.200 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T18:20:46.201 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T18:20:46.202 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T18:20:46.203 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T18:20:46.204 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T18:20:46.205 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T18:20:46.206 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T18:20:46.207 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T18:20:46.208 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T18:20:46.209 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T18:20:46.210 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T18:20:46.211 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T18:20:46.212 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T18:20:46.213 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T18:20:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T18:20:46.215 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T18:20:46.216 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T18:20:46.217 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T18:20:46.218 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T18:20:46.219 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T18:20:46.220 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T18:20:46.221 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T18:20:46.222 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T18:20:46.223 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T18:20:46.224 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T18:20:46.225 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T18:20:46.226 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T18:20:46.227 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T18:20:46.228 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T18:20:46.229 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T18:20:46.230 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T18:20:46.231 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T18:20:46.232 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T18:20:46.233 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T18:20:46.234 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T18:20:46.235 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T18:20:46.236 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T18:20:46.237 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T18:20:46.238 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T18:20:46.239 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T18:20:46.240 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T18:20:46.241 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T18:20:46.242 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T18:20:46.243 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T18:20:46.244 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T18:20:46.245 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T18:20:46.246 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T18:20:46.247 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T18:20:46.248 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T18:20:46.249 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T18:20:46.250 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T18:20:46.251 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T18:20:46.252 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T18:20:46.253 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T18:20:46.254 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T18:20:46.255 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T18:20:46.256 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T18:20:46.257 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T18:20:46.258 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T18:20:46.259 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T18:20:46.260 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T18:20:46.261 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T18:20:46.262 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T18:20:46.263 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T18:20:46.264 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T18:20:46.265 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T18:20:46.266 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T18:20:46.267 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T18:20:46.268 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T18:20:46.269 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T18:20:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T18:20:46.271 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T18:20:46.272 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T18:20:46.273 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T18:20:46.274 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T18:20:46.275 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T18:20:46.276 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T18:20:46.277 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T18:20:46.278 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T18:20:46.279 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T18:20:46.280 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T18:20:46.281 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T18:20:46.282 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T18:20:46.283 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T18:20:46.284 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T18:20:46.285 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T18:20:46.286 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T18:20:46.287 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T18:20:46.288 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T18:20:46.289 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T18:20:46.290 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T18:20:46.291 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T18:20:46.292 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T18:20:46.293 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T18:20:46.294 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T18:20:46.295 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T18:20:46.296 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T18:20:46.296 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T18:20:46.296 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T18:20:46.296 INFO:teuthology.orchestra.run.smithi063.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T18:20:46.300 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.300 DEBUG:teuthology.orchestra.run.smithi063:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T18:20:46.323 INFO:teuthology.orchestra.run.smithi063.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T18:20:46.323 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T18:20:46.323 INFO:teuthology.orchestra.run.smithi063.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T18:20:46.323 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:for file in vmlinuz System.map config; do 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: fi 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:done 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T18:20:46.324 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.325 INFO:teuthology.orchestra.run.smithi063.stdout:fi 2024-04-25T18:20:46.325 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 68.el9 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-04-25T18:20:46.405 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8264254 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:20:46.406 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T18:20:46.407 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T18:20:46.407 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T18:20:46.407 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-04-25T18:20:46.407 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:20:46.407 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T18:20:46.408 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T18:20:46.408 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:46.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T18:20:46.957 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-04-25T18:20:48.365 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-25T18:20:48.396 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-04-25T18:20:48.398 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-04-25T18:20:48.430 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.dX3unaaKoF 2024-04-25T18:20:48.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dX3unaaKoF 2024-04-25T18:20:48.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.dX3unaaKoF 2024-04-25T18:20:48.646 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.dX3unaaKoF is 6KB 2024-04-25T18:20:48.696 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.dX3unaaKoF 2024-04-25T18:20:48.711 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T18:20:48.773 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-25T18:20:48.773 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-25T18:20:48.773 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-25T18:20:48.773 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T18:20:48.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-25T18:20:48.923 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-04-25T18:21:18.926 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-25T18:21:18.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:21:18.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:21:37.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-25T18:21:46.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:21:46.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:21:49.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-25T18:22:01.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:22:01.504 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:23:01.507 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T18:23:16.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:23:16.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:23:16.860 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-25T18:23:17.551 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:23:17.551 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T18:23:17.552 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-04-25T18:23:17.608 INFO:teuthology.orchestra.run.smithi063.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T18:23:17.608 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T18:23:17.609 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T18:23:17.609 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T18:23:18.610 DEBUG:teuthology.parallel:result is None 2024-04-25T18:23:18.610 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T18:23:18.617 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T18:23:18.618 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T18:23:18.626 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T18:23:18.628 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T18:23:18.678 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T18:23:18.684 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T18:23:18.690 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T18:23:18.691 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T18:23:18.699 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T18:23:18.701 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T18:23:18.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T18:23:18.815 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T18:23:18.815 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T18:23:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:23:18.832 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T18:23:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:23:18.849 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T18:23:18.855 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T18:23:18.876 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T18:23:18.915 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:18.931 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:18.949 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:18.964 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:18.979 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:19.058 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T18:23:19.064 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T18:23:19.074 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T18:23:19.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T18:23:19.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T18:23:19.088 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T18:23:19.170 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T18:23:19.180 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T18:23:19.180 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T18:23:19.187 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T18:23:19.189 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T18:23:19.236 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:19.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:19.419 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:19.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:19.595 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:23:19.595 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T18:23:19.681 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:19.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:19.789 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:19.833 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:19.873 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:23:19.874 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T18:23:19.939 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:19.984 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T18:23:20.032 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:20.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T18:23:20.157 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:23:20.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T18:23:20.223 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-04-25T18:23:20.231 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-25T18:23:20.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-04-25T18:23:20.268 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T18:23:20.298 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T18:23:20.341 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T18:23:20.743 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T18:23:20.751 INFO:teuthology.task.internal:Starting timer... 2024-04-25T18:23:20.751 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T18:23:20.762 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T18:23:20.771 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T18:23:20.772 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-04-25T18:23:20.883 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-04-25T18:23:20.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-25T18:23:20.959 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-25T18:23:20.961 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-04-25T18:23:21.023 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-04-25T18:23:21.025 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T18:23:21.025 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-04-25T18:23:21.087 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-04-25T18:23:21.088 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-25T18:23:21.116 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-25T18:23:21.116 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-04-25T18:23:21.137 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-04-25T18:23:21.137 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive'} 2024-04-25T18:23:21.137 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T18:23:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:23:21.222 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T18:23:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:23:21.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T18:23:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:23:21.277 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T18:23:21.277 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-04-25T18:23:21.357 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-25T18:23:21.384 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-04-25T18:23:21.408 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T18:23:21.419 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T18:23:21.424 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T18:23:21.547 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T18:23:21.562 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T18:23:21.563 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi070.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T18:28:33.206 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2024-04-25T18:28:33.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:28:33.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:28:33.316 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-25T18:28:33.468 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T18:28:33.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-25T18:28:33.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:28:33.541 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-25T18:28:33.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-25T18:28:33.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-25T18:28:33.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T18:28:33.692 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-04-25T18:28:33.771 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-04-25T18:28:33.772 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T18:28:33.783 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T18:28:33.783 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T18:28:33.784 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T18:28:33.791 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T18:28:33.791 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T18:28:33.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T18:28:33.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T18:28:33.823 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T18:28:33.837 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T18:28:33.842 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T18:28:33.856 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T18:28:33.861 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-04-25T18:28:33.872 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-04-25T18:28:33.879 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T18:28:33.880 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-04-25T18:28:33.885 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T18:28:33.891 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-04-25T18:28:33.898 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T18:28:33.904 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T18:28:33.918 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T18:28:33.922 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T18:28:33.954 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.958 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.982 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:33.985 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:34.002 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-04-25T18:28:34.035 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-04-25T18:28:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T18:28:34.125 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T18:28:34.265 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-04-25T18:28:34.274 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T18:28:34.274 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-25T18:28:34.275 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:34.275 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:34.275 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:34.275 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T18:28:34.276 INFO:teuthology.run_tasks:Running task install... 2024-04-25T18:28:34.288 DEBUG:teuthology.task.install:project ceph 2024-04-25T18:28:34.289 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'}} 2024-04-25T18:28:34.289 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f'} 2024-04-25T18:28:34.289 INFO:teuthology.task.install:Using flavor: default 2024-04-25T18:28:34.294 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T18:28:34.295 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T18:28:34.295 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T18:28:34.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:28:34.297 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T18:28:34.298 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:28:34.299 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'tag': None, 'wait_for_package': False} 2024-04-25T18:28:34.299 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:28:34.443 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T18:28:34.444 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T18:28:34.470 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T18:28:34.470 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T18:28:34.517 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/ 2024-04-25T18:28:34.517 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3156.gd69f13d8 2024-04-25T18:28:34.776 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T18:28:34.776 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:28:34.776 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T18:28:34.805 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T18:28:34.805 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T18:28:34.810 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T18:28:34.810 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:28:34.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T18:28:34.839 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T18:28:34.839 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T18:28:34.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T18:28:34.892 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T18:28:34.892 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:28:34.892 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T18:28:34.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T18:28:34.951 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T18:28:34.983 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T18:28:34.990 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T18:28:34.990 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d69f13d827952bb07bb9b7973bca14f97d764d6f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T18:28:35.016 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-04-25T18:28:35.018 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-04-25T18:28:35.048 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-04-25T18:28:35.050 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-04-25T18:28:35.125 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T18:28:35.245 INFO:teuthology.orchestra.run.smithi114.stdout:48 files removed 2024-04-25T18:28:35.247 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T18:28:35.272 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T18:28:35.274 INFO:teuthology.orchestra.run.smithi070.stdout:48 files removed 2024-04-25T18:28:35.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T18:28:35.326 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-04-25T18:28:35.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-04-25T18:28:35.667 INFO:teuthology.orchestra.run.smithi063.stdout:48 files removed 2024-04-25T18:28:35.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T18:28:36.648 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 75 kB/s | 83 kB 00:01 2024-04-25T18:28:36.808 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 67 kB/s | 83 kB 00:01 2024-04-25T18:28:37.099 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 89 kB/s | 83 kB 00:00 2024-04-25T18:28:37.256 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-04-25T18:28:37.525 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 17 kB/s | 12 kB 00:00 2024-04-25T18:28:37.974 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-04-25T18:28:38.191 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 12 kB/s | 12 kB 00:00 2024-04-25T18:28:38.226 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-25T18:28:38.862 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-25T18:28:38.884 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-04-25T18:28:39.125 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-04-25T18:28:40.070 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-04-25T18:28:41.013 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T18:28:41.136 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T18:28:42.329 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-25T18:28:45.071 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-25T18:28:45.145 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-25T18:28:46.661 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-25T18:28:46.776 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-25T18:28:47.615 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.2 MB 00:01 2024-04-25T18:28:47.913 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T18:28:49.594 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 21 MB 00:02 2024-04-25T18:28:49.751 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-25T18:28:52.727 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-04-25T18:28:53.037 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-25T18:28:54.553 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T18:28:55.197 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:28:55.198 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:28:55.209 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T18:28:55.210 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T18:28:55.210 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T18:28:55.211 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T18:28:55.213 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T18:28:55.215 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T18:28:55.216 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T18:28:55.277 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-25T18:28:55.297 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T18:28:55.298 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T18:28:55.299 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T18:28:55.300 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T18:28:55.301 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T18:28:55.302 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T18:28:55.303 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T18:28:55.304 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T18:28:55.305 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T18:28:55.306 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T18:28:55.307 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T18:28:55.308 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-04-25T18:28:55.309 INFO:teuthology.orchestra.run.smithi070.stdout:Install 148 Packages 2024-04-25T18:28:55.310 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 4 Packages 2024-04-25T18:28:55.310 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:28:55.310 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 222 M 2024-04-25T18:28:55.310 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-25T18:28:56.130 INFO:teuthology.orchestra.run.smithi070.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T18:28:56.355 INFO:teuthology.orchestra.run.smithi070.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 3.9 MB/s | 890 kB 00:00 2024-04-25T18:28:56.405 INFO:teuthology.orchestra.run.smithi070.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 2.8 MB/s | 145 kB 00:00 2024-04-25T18:28:56.662 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:28:56.663 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:28:56.674 INFO:teuthology.orchestra.run.smithi114.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T18:28:56.675 INFO:teuthology.orchestra.run.smithi114.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T18:28:56.676 INFO:teuthology.orchestra.run.smithi114.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T18:28:56.676 INFO:teuthology.orchestra.run.smithi114.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T18:28:56.678 INFO:teuthology.orchestra.run.smithi114.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T18:28:56.680 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T18:28:56.681 INFO:teuthology.orchestra.run.smithi114.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T18:28:56.739 INFO:teuthology.orchestra.run.smithi070.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 7.1 MB/s | 2.4 MB 00:00 2024-04-25T18:28:56.743 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T18:28:56.763 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T18:28:56.764 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:28:56.765 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T18:28:56.766 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T18:28:56.767 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T18:28:56.768 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T18:28:56.769 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T18:28:56.770 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T18:28:56.771 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T18:28:56.772 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T18:28:56.773 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T18:28:56.774 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================= 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout:Install 148 Packages 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 4 Packages 2024-04-25T18:28:56.775 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:28:56.776 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 222 M 2024-04-25T18:28:56.776 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-04-25T18:28:56.823 INFO:teuthology.orchestra.run.smithi070.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 6.2 MB/s | 5.4 MB 00:00 2024-04-25T18:28:56.989 INFO:teuthology.orchestra.run.smithi070.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 6.0 MB/s | 1.5 MB 00:00 2024-04-25T18:28:57.323 INFO:teuthology.orchestra.run.smithi070.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 9.4 MB/s | 4.7 MB 00:00 2024-04-25T18:28:58.104 INFO:teuthology.orchestra.run.smithi114.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 9.8 kB/s | 6.6 kB 00:00 2024-04-25T18:28:58.757 INFO:teuthology.orchestra.run.smithi114.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.3 MB/s | 890 kB 00:00 2024-04-25T18:28:58.840 INFO:teuthology.orchestra.run.smithi070.stdout:(8/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 7.0 MB/s | 11 MB 00:01 2024-04-25T18:28:58.846 INFO:teuthology.orchestra.run.smithi114.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 1.5 MB/s | 145 kB 00:00 2024-04-25T18:28:58.907 INFO:teuthology.orchestra.run.smithi070.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 379 kB/s | 25 kB 00:00 2024-04-25T18:28:58.910 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-25T18:28:59.141 INFO:teuthology.orchestra.run.smithi070.stdout:(10/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 7.8 MB/s | 17 MB 00:02 2024-04-25T18:28:59.249 INFO:teuthology.orchestra.run.smithi070.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.0 MB/s | 225 kB 00:00 2024-04-25T18:28:59.341 INFO:teuthology.orchestra.run.smithi070.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 295 kB/s | 27 kB 00:00 2024-04-25T18:28:59.425 INFO:teuthology.orchestra.run.smithi070.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 380 kB/s | 32 kB 00:00 2024-04-25T18:28:59.592 INFO:teuthology.orchestra.run.smithi070.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.4 MB/s | 747 kB 00:00 2024-04-25T18:28:59.684 INFO:teuthology.orchestra.run.smithi070.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.7 MB/s | 163 kB 00:00 2024-04-25T18:28:59.759 INFO:teuthology.orchestra.run.smithi070.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.6 MB/s | 126 kB 00:00 2024-04-25T18:28:59.813 INFO:teuthology.orchestra.run.smithi114.stdout:(4/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 2.4 MB/s | 2.4 MB 00:00 2024-04-25T18:28:59.851 INFO:teuthology.orchestra.run.smithi070.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 5.3 MB/s | 500 kB 00:00 2024-04-25T18:29:00.063 INFO:teuthology.orchestra.run.smithi114.stdout:(5/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 2.0 MB/s | 5.4 MB 00:02 2024-04-25T18:29:00.488 INFO:teuthology.orchestra.run.smithi114.stdout:(6/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 1.5 MB 00:00 2024-04-25T18:29:00.776 INFO:teuthology.orchestra.run.smithi070.stdout:(18/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 5.8 MB/s | 5.4 MB 00:00 2024-04-25T18:29:00.877 INFO:teuthology.orchestra.run.smithi070.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 452 kB/s | 45 kB 00:00 2024-04-25T18:29:01.093 INFO:teuthology.orchestra.run.smithi070.stdout:(20/152): ceph-common-19.0.0-3156.gd69f13d8.el7 4.1 MB/s | 21 MB 00:05 2024-04-25T18:29:01.119 INFO:teuthology.orchestra.run.smithi070.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 557 kB/s | 135 kB 00:00 2024-04-25T18:29:01.186 INFO:teuthology.orchestra.run.smithi070.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.7 MB/s | 161 kB 00:00 2024-04-25T18:29:01.211 INFO:teuthology.orchestra.run.smithi070.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.4 MB/s | 320 kB 00:00 2024-04-25T18:29:01.294 INFO:teuthology.orchestra.run.smithi070.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.2 MB/s | 99 kB 00:00 2024-04-25T18:29:01.345 INFO:teuthology.orchestra.run.smithi070.stdout:(25/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.8 MB/s | 298 kB 00:00 2024-04-25T18:29:01.370 INFO:teuthology.orchestra.run.smithi070.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.1 MB/s | 85 kB 00:00 2024-04-25T18:29:01.397 INFO:teuthology.orchestra.run.smithi114.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 3.5 MB/s | 4.7 MB 00:01 2024-04-25T18:29:01.414 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:29:01.415 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T18:29:01.437 INFO:teuthology.orchestra.run.smithi063.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T18:29:01.438 INFO:teuthology.orchestra.run.smithi063.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T18:29:01.440 INFO:teuthology.orchestra.run.smithi063.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T18:29:01.441 INFO:teuthology.orchestra.run.smithi063.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T18:29:01.443 INFO:teuthology.orchestra.run.smithi063.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T18:29:01.445 INFO:teuthology.orchestra.run.smithi070.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.2 MB/s | 170 kB 00:00 2024-04-25T18:29:01.447 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T18:29:01.448 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T18:29:01.524 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-25T18:29:01.529 INFO:teuthology.orchestra.run.smithi070.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 410 kB/s | 34 kB 00:00 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 6.6 k 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:29:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 890 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 145 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 1.5 M 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 142 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 3.5 M 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 7.4 M 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 49 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 11 M 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 49 M 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 273 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 762 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 225 k 2024-04-25T18:29:01.543 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 27 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 18 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 32 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 747 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 126 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T18:29:01.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 161 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 320 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 298 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 99 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 85 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 170 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.4 M 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 3.1 M 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-25T18:29:01.545 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 21 M 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 34 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 2.4 M 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 249 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 4.7 M 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 17 M 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3156.gd69f13d8.el7 ceph-noarch 16 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 25 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T18:29:01.546 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 163 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 500 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T18:29:01.547 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 5.4 M 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T18:29:01.548 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 45 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-3156.gd69f13d8.el7 ceph 135 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T18:29:01.549 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T18:29:01.550 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T18:29:01.551 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T18:29:01.552 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T18:29:01.553 INFO:teuthology.orchestra.run.smithi063.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout:Install 148 Packages 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 4 Packages 2024-04-25T18:29:01.554 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:29:01.555 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 222 M 2024-04-25T18:29:01.555 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-25T18:29:01.621 INFO:teuthology.orchestra.run.smithi070.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.5 MB/s | 142 kB 00:00 2024-04-25T18:29:02.329 INFO:teuthology.orchestra.run.smithi070.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 3.1 MB/s | 3.1 MB 00:00 2024-04-25T18:29:02.388 INFO:teuthology.orchestra.run.smithi070.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 4.6 MB/s | 3.5 MB 00:00 2024-04-25T18:29:02.480 INFO:teuthology.orchestra.run.smithi070.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 2.7 MB/s | 249 kB 00:00 2024-04-25T18:29:02.555 INFO:teuthology.orchestra.run.smithi070.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 659 kB/s | 49 kB 00:00 2024-04-25T18:29:02.630 INFO:teuthology.orchestra.run.smithi070.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 219 kB/s | 16 kB 00:00 2024-04-25T18:29:02.714 INFO:teuthology.orchestra.run.smithi070.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 273 kB 00:00 2024-04-25T18:29:02.870 INFO:teuthology.orchestra.run.smithi063.stdout:(1/152): ceph-19.0.0-3156.gd69f13d8.el7.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-25T18:29:02.889 INFO:teuthology.orchestra.run.smithi070.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 4.3 MB/s | 762 kB 00:00 2024-04-25T18:29:02.939 INFO:teuthology.orchestra.run.smithi070.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 366 kB/s | 18 kB 00:00 2024-04-25T18:29:02.955 INFO:teuthology.orchestra.run.smithi114.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 3.8 MB/s | 21 MB 00:05 2024-04-25T18:29:03.039 INFO:teuthology.orchestra.run.smithi114.stdout:(9/152): ceph-selinux-19.0.0-3156.gd69f13d8.el7 303 kB/s | 25 kB 00:00 2024-04-25T18:29:03.040 INFO:teuthology.orchestra.run.smithi070.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 803 kB/s | 80 kB 00:00 2024-04-25T18:29:03.123 INFO:teuthology.orchestra.run.smithi070.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 803 kB 00:00 2024-04-25T18:29:03.165 INFO:teuthology.orchestra.run.smithi070.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-25T18:29:03.207 INFO:teuthology.orchestra.run.smithi070.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-25T18:29:03.266 INFO:teuthology.orchestra.run.smithi070.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-25T18:29:03.300 INFO:teuthology.orchestra.run.smithi070.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T18:29:03.350 INFO:teuthology.orchestra.run.smithi070.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T18:29:03.392 INFO:teuthology.orchestra.run.smithi070.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T18:29:03.434 INFO:teuthology.orchestra.run.smithi070.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-04-25T18:29:03.518 INFO:teuthology.orchestra.run.smithi070.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 16 MB/s | 689 kB 00:00 2024-04-25T18:29:03.534 INFO:teuthology.orchestra.run.smithi070.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T18:29:03.568 INFO:teuthology.orchestra.run.smithi070.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 107 kB 00:00 2024-04-25T18:29:03.602 INFO:teuthology.orchestra.run.smithi070.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 645 kB/s | 21 kB 00:00 2024-04-25T18:29:03.644 INFO:teuthology.orchestra.run.smithi070.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T18:29:03.677 INFO:teuthology.orchestra.run.smithi070.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 977 kB/s | 33 kB 00:00 2024-04-25T18:29:03.786 INFO:teuthology.orchestra.run.smithi070.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-25T18:29:03.828 INFO:teuthology.orchestra.run.smithi070.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 437 kB/s | 18 kB 00:00 2024-04-25T18:29:03.861 INFO:teuthology.orchestra.run.smithi070.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-25T18:29:03.903 INFO:teuthology.orchestra.run.smithi070.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-25T18:29:03.904 INFO:teuthology.orchestra.run.smithi063.stdout:(2/152): ceph-fuse-19.0.0-3156.gd69f13d8.el7.x8 862 kB/s | 890 kB 00:01 2024-04-25T18:29:03.937 INFO:teuthology.orchestra.run.smithi070.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T18:29:03.996 INFO:teuthology.orchestra.run.smithi070.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T18:29:04.038 INFO:teuthology.orchestra.run.smithi070.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 582 kB 00:00 2024-04-25T18:29:04.071 INFO:teuthology.orchestra.run.smithi070.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 768 kB/s | 25 kB 00:00 2024-04-25T18:29:04.095 INFO:teuthology.orchestra.run.smithi063.stdout:(3/152): ceph-immutable-object-cache-19.0.0-315 759 kB/s | 145 kB 00:00 2024-04-25T18:29:04.105 INFO:teuthology.orchestra.run.smithi070.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T18:29:04.173 INFO:teuthology.orchestra.run.smithi070.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 292 kB 00:00 2024-04-25T18:29:04.214 INFO:teuthology.orchestra.run.smithi070.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 4.5 MB/s | 188 kB 00:00 2024-04-25T18:29:04.248 INFO:teuthology.orchestra.run.smithi070.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-04-25T18:29:04.406 INFO:teuthology.orchestra.run.smithi070.stdout:(65/152): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-04-25T18:29:04.490 INFO:teuthology.orchestra.run.smithi070.stdout:(66/152): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-04-25T18:29:04.582 INFO:teuthology.orchestra.run.smithi070.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-04-25T18:29:04.624 INFO:teuthology.orchestra.run.smithi070.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T18:29:04.699 INFO:teuthology.orchestra.run.smithi070.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.8 MB/s | 911 kB 00:00 2024-04-25T18:29:04.724 INFO:teuthology.orchestra.run.smithi070.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-25T18:29:04.826 INFO:teuthology.orchestra.run.smithi070.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.9 MB/s | 1.0 MB 00:00 2024-04-25T18:29:04.919 INFO:teuthology.orchestra.run.smithi070.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 26 MB/s | 5.8 MB 00:00 2024-04-25T18:29:04.943 INFO:teuthology.orchestra.run.smithi070.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 764 kB/s | 87 kB 00:00 2024-04-25T18:29:04.946 INFO:teuthology.orchestra.run.smithi063.stdout:(4/152): ceph-base-19.0.0-3156.gd69f13d8.el7.x8 2.2 MB/s | 5.4 MB 00:02 2024-04-25T18:29:05.119 INFO:teuthology.orchestra.run.smithi070.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 6.0 MB 00:00 2024-04-25T18:29:05.131 INFO:teuthology.orchestra.run.smithi114.stdout:(10/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el 2.8 MB/s | 11 MB 00:03 2024-04-25T18:29:05.144 INFO:teuthology.orchestra.run.smithi070.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-25T18:29:05.169 INFO:teuthology.orchestra.run.smithi070.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T18:29:05.195 INFO:teuthology.orchestra.run.smithi070.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-25T18:29:05.220 INFO:teuthology.orchestra.run.smithi070.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T18:29:05.245 INFO:teuthology.orchestra.run.smithi070.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-25T18:29:05.270 INFO:teuthology.orchestra.run.smithi070.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-25T18:29:05.321 INFO:teuthology.orchestra.run.smithi063.stdout:(5/152): ceph-mgr-19.0.0-3156.gd69f13d8.el7.x86 4.0 MB/s | 1.5 MB 00:00 2024-04-25T18:29:05.322 INFO:teuthology.orchestra.run.smithi114.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 1.1 MB/s | 225 kB 00:00 2024-04-25T18:29:05.337 INFO:teuthology.orchestra.run.smithi070.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-25T18:29:05.441 INFO:teuthology.orchestra.run.smithi063.stdout:(6/152): ceph-mds-19.0.0-3156.gd69f13d8.el7.x86 1.8 MB/s | 2.4 MB 00:01 2024-04-25T18:29:05.457 INFO:teuthology.orchestra.run.smithi114.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 202 kB/s | 27 kB 00:00 2024-04-25T18:29:05.496 INFO:teuthology.orchestra.run.smithi070.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-04-25T18:29:05.522 INFO:teuthology.orchestra.run.smithi070.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 222 kB 00:00 2024-04-25T18:29:05.547 INFO:teuthology.orchestra.run.smithi070.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T18:29:05.557 INFO:teuthology.orchestra.run.smithi114.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 317 kB/s | 32 kB 00:00 2024-04-25T18:29:05.572 INFO:teuthology.orchestra.run.smithi070.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-25T18:29:05.606 INFO:teuthology.orchestra.run.smithi070.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-04-25T18:29:05.631 INFO:teuthology.orchestra.run.smithi070.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-04-25T18:29:05.664 INFO:teuthology.orchestra.run.smithi070.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-25T18:29:05.690 INFO:teuthology.orchestra.run.smithi070.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-04-25T18:29:05.715 INFO:teuthology.orchestra.run.smithi070.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T18:29:05.790 INFO:teuthology.orchestra.run.smithi070.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 557 kB/s | 42 kB 00:00 2024-04-25T18:29:05.966 INFO:teuthology.orchestra.run.smithi114.stdout:(14/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 3.1 MB/s | 17 MB 00:05 2024-04-25T18:29:05.982 INFO:teuthology.orchestra.run.smithi070.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 231 kB/s | 44 kB 00:00 2024-04-25T18:29:06.007 INFO:teuthology.orchestra.run.smithi114.stdout:(15/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 1.6 MB/s | 747 kB 00:00 2024-04-25T18:29:06.091 INFO:teuthology.orchestra.run.smithi114.stdout:(16/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.3 MB/s | 163 kB 00:00 2024-04-25T18:29:06.141 INFO:teuthology.orchestra.run.smithi114.stdout:(17/152): librados-devel-19.0.0-3156.gd69f13d8. 949 kB/s | 126 kB 00:00 2024-04-25T18:29:06.222 INFO:teuthology.orchestra.run.smithi063.stdout:(7/152): ceph-mon-19.0.0-3156.gd69f13d8.el7.x86 5.2 MB/s | 4.7 MB 00:00 2024-04-25T18:29:06.325 INFO:teuthology.orchestra.run.smithi114.stdout:(18/152): libradosstriper1-19.0.0-3156.gd69f13d 2.1 MB/s | 500 kB 00:00 2024-04-25T18:29:06.433 INFO:teuthology.orchestra.run.smithi070.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-25T18:29:06.433 INFO:teuthology.orchestra.run.smithi114.stdout:(19/152): python3-ceph-argparse-19.0.0-3156.gd6 416 kB/s | 45 kB 00:00 2024-04-25T18:29:06.458 INFO:teuthology.orchestra.run.smithi070.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 466 kB/s | 221 kB 00:00 2024-04-25T18:29:06.483 INFO:teuthology.orchestra.run.smithi070.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 721 kB/s | 36 kB 00:00 2024-04-25T18:29:06.500 INFO:teuthology.orchestra.run.smithi070.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-25T18:29:06.525 INFO:teuthology.orchestra.run.smithi070.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2024-04-25T18:29:06.550 INFO:teuthology.orchestra.run.smithi070.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 305 kB 00:00 2024-04-25T18:29:06.576 INFO:teuthology.orchestra.run.smithi070.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T18:29:06.610 INFO:teuthology.orchestra.run.smithi070.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T18:29:06.626 INFO:teuthology.orchestra.run.smithi070.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 889 kB/s | 45 kB 00:00 2024-04-25T18:29:06.643 INFO:teuthology.orchestra.run.smithi070.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 666 kB/s | 22 kB 00:00 2024-04-25T18:29:06.702 INFO:teuthology.orchestra.run.smithi070.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T18:29:07.259 INFO:teuthology.orchestra.run.smithi114.stdout:(20/152): python3-ceph-common-19.0.0-3156.gd69f 163 kB/s | 135 kB 00:00 2024-04-25T18:29:07.369 INFO:teuthology.orchestra.run.smithi070.stdout:(104/152): ceph-test-19.0.0-3156.gd69f13d8.el7. 5.8 MB/s | 49 MB 00:08 2024-04-25T18:29:07.601 INFO:teuthology.orchestra.run.smithi114.stdout:(21/152): python3-cephfs-19.0.0-3156.gd69f13d8. 472 kB/s | 161 kB 00:00 2024-04-25T18:29:07.781 INFO:teuthology.orchestra.run.smithi063.stdout:(8/152): ceph-common-19.0.0-3156.gd69f13d8.el7. 4.0 MB/s | 21 MB 00:05 2024-04-25T18:29:07.819 INFO:teuthology.orchestra.run.smithi070.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 43 kB/s | 19 kB 00:00 2024-04-25T18:29:07.851 INFO:teuthology.orchestra.run.smithi114.stdout:(22/152): python3-rados-19.0.0-3156.gd69f13d8.e 1.3 MB/s | 320 kB 00:00 2024-04-25T18:29:07.953 INFO:teuthology.orchestra.run.smithi070.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 247 kB/s | 309 kB 00:01 2024-04-25T18:29:07.981 INFO:teuthology.orchestra.run.smithi063.stdout:(9/152): ceph-radosgw-19.0.0-3156.gd69f13d8.el7 6.0 MB/s | 11 MB 00:01 2024-04-25T18:29:08.001 INFO:teuthology.orchestra.run.smithi114.stdout:(23/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 2.9 MB/s | 5.4 MB 00:01 2024-04-25T18:29:08.006 INFO:teuthology.orchestra.run.smithi063.stdout:(10/152): ceph-selinux-19.0.0-3156.gd69f13d8.el 112 kB/s | 25 kB 00:00 2024-04-25T18:29:08.053 INFO:teuthology.orchestra.run.smithi070.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 389 kB/s | 548 kB 00:01 2024-04-25T18:29:08.076 INFO:teuthology.orchestra.run.smithi114.stdout:(24/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 1.3 MB/s | 99 kB 00:00 2024-04-25T18:29:08.078 INFO:teuthology.orchestra.run.smithi070.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-04-25T18:29:08.098 INFO:teuthology.orchestra.run.smithi063.stdout:(11/152): cephfs-mirror-19.0.0-3156.gd69f13d8.e 2.4 MB/s | 225 kB 00:00 2024-04-25T18:29:08.110 INFO:teuthology.orchestra.run.smithi114.stdout:(25/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 1.1 MB/s | 298 kB 00:00 2024-04-25T18:29:08.144 INFO:teuthology.orchestra.run.smithi114.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 1.2 MB/s | 85 kB 00:00 2024-04-25T18:29:08.162 INFO:teuthology.orchestra.run.smithi070.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-25T18:29:08.208 INFO:teuthology.orchestra.run.smithi063.stdout:(12/152): cephfs-shell-19.0.0-3156.gd69f13d8.el 246 kB/s | 27 kB 00:00 2024-04-25T18:29:08.227 INFO:teuthology.orchestra.run.smithi114.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 2.0 MB/s | 170 kB 00:00 2024-04-25T18:29:08.270 INFO:teuthology.orchestra.run.smithi070.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-04-25T18:29:08.306 INFO:teuthology.orchestra.run.smithi063.stdout:(13/152): libcephfs-devel-19.0.0-3156.gd69f13d8 326 kB/s | 32 kB 00:00 2024-04-25T18:29:08.319 INFO:teuthology.orchestra.run.smithi114.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 374 kB/s | 34 kB 00:00 2024-04-25T18:29:08.428 INFO:teuthology.orchestra.run.smithi114.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-25T18:29:08.462 INFO:teuthology.orchestra.run.smithi070.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 503 kB/s | 151 kB 00:00 2024-04-25T18:29:08.482 INFO:teuthology.orchestra.run.smithi063.stdout:(14/152): libcephfs2-19.0.0-3156.gd69f13d8.el7. 4.2 MB/s | 747 kB 00:00 2024-04-25T18:29:08.590 INFO:teuthology.orchestra.run.smithi063.stdout:(15/152): libcephsqlite-19.0.0-3156.gd69f13d8.e 1.5 MB/s | 163 kB 00:00 2024-04-25T18:29:08.682 INFO:teuthology.orchestra.run.smithi063.stdout:(16/152): librados-devel-19.0.0-3156.gd69f13d8. 1.4 MB/s | 126 kB 00:00 2024-04-25T18:29:08.808 INFO:teuthology.orchestra.run.smithi063.stdout:(17/152): libradosstriper1-19.0.0-3156.gd69f13d 3.9 MB/s | 500 kB 00:00 2024-04-25T18:29:09.346 INFO:teuthology.orchestra.run.smithi070.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 620 kB/s | 548 kB 00:00 2024-04-25T18:29:09.479 INFO:teuthology.orchestra.run.smithi070.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 326 kB/s | 43 kB 00:00 2024-04-25T18:29:09.595 INFO:teuthology.orchestra.run.smithi063.stdout:(18/152): ceph-osd-19.0.0-3156.gd69f13d8.el7.x8 4.1 MB/s | 17 MB 00:04 2024-04-25T18:29:09.604 INFO:teuthology.orchestra.run.smithi070.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-04-25T18:29:09.630 INFO:teuthology.orchestra.run.smithi070.stdout:(115/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 617 kB/s | 838 kB 00:01 2024-04-25T18:29:09.653 INFO:teuthology.orchestra.run.smithi114.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 2.9 MB/s | 3.5 MB 00:01 2024-04-25T18:29:09.705 INFO:teuthology.orchestra.run.smithi070.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-25T18:29:09.750 INFO:teuthology.orchestra.run.smithi063.stdout:(19/152): librgw2-19.0.0-3156.gd69f13d8.el7.x86 5.7 MB/s | 5.4 MB 00:00 2024-04-25T18:29:09.775 INFO:teuthology.orchestra.run.smithi063.stdout:(20/152): python3-ceph-argparse-19.0.0-3156.gd6 251 kB/s | 45 kB 00:00 2024-04-25T18:29:09.828 INFO:teuthology.orchestra.run.smithi114.stdout:(31/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 1.8 MB/s | 3.1 MB 00:01 2024-04-25T18:29:09.833 INFO:teuthology.orchestra.run.smithi063.stdout:(21/152): python3-ceph-common-19.0.0-3156.gd69f 1.6 MB/s | 135 kB 00:00 2024-04-25T18:29:09.867 INFO:teuthology.orchestra.run.smithi063.stdout:(22/152): python3-cephfs-19.0.0-3156.gd69f13d8. 1.7 MB/s | 161 kB 00:00 2024-04-25T18:29:09.905 INFO:teuthology.orchestra.run.smithi070.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 624 kB/s | 172 kB 00:00 2024-04-25T18:29:09.926 INFO:teuthology.orchestra.run.smithi063.stdout:(23/152): python3-rados-19.0.0-3156.gd69f13d8.e 3.4 MB/s | 320 kB 00:00 2024-04-25T18:29:09.984 INFO:teuthology.orchestra.run.smithi063.stdout:(24/152): python3-rbd-19.0.0-3156.gd69f13d8.el7 2.5 MB/s | 298 kB 00:00 2024-04-25T18:29:09.987 INFO:teuthology.orchestra.run.smithi114.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.5 MB/s | 249 kB 00:00 2024-04-25T18:29:10.027 INFO:teuthology.orchestra.run.smithi063.stdout:(25/152): python3-rgw-19.0.0-3156.gd69f13d8.el7 996 kB/s | 99 kB 00:00 2024-04-25T18:29:10.071 INFO:teuthology.orchestra.run.smithi114.stdout:(33/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 591 kB/s | 49 kB 00:00 2024-04-25T18:29:10.077 INFO:teuthology.orchestra.run.smithi063.stdout:(26/152): rbd-fuse-19.0.0-3156.gd69f13d8.el7.x8 935 kB/s | 85 kB 00:00 2024-04-25T18:29:10.154 INFO:teuthology.orchestra.run.smithi114.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3156.gd 198 kB/s | 16 kB 00:00 2024-04-25T18:29:10.185 INFO:teuthology.orchestra.run.smithi063.stdout:(27/152): rbd-nbd-19.0.0-3156.gd69f13d8.el7.x86 1.5 MB/s | 170 kB 00:00 2024-04-25T18:29:10.214 INFO:teuthology.orchestra.run.smithi070.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-25T18:29:10.315 INFO:teuthology.orchestra.run.smithi063.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3156.g 266 kB/s | 34 kB 00:00 2024-04-25T18:29:10.322 INFO:teuthology.orchestra.run.smithi070.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-25T18:29:10.338 INFO:teuthology.orchestra.run.smithi114.stdout:(35/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 1.5 MB/s | 273 kB 00:00 2024-04-25T18:29:10.348 INFO:teuthology.orchestra.run.smithi070.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 685 kB/s | 303 kB 00:00 2024-04-25T18:29:10.419 INFO:teuthology.orchestra.run.smithi063.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3156.gd69f13d 1.3 MB/s | 142 kB 00:00 2024-04-25T18:29:10.511 INFO:teuthology.orchestra.run.smithi063.stdout:(30/152): rbd-mirror-19.0.0-3156.gd69f13d8.el7. 6.4 MB/s | 3.1 MB 00:00 2024-04-25T18:29:10.698 INFO:teuthology.orchestra.run.smithi070.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 548 kB/s | 206 kB 00:00 2024-04-25T18:29:10.730 INFO:teuthology.orchestra.run.smithi114.stdout:(36/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 1.9 MB/s | 762 kB 00:00 2024-04-25T18:29:10.838 INFO:teuthology.orchestra.run.smithi114.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 169 kB/s | 18 kB 00:00 2024-04-25T18:29:10.955 INFO:teuthology.orchestra.run.smithi114.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 689 kB/s | 80 kB 00:00 2024-04-25T18:29:10.990 INFO:teuthology.orchestra.run.smithi070.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 495 kB/s | 144 kB 00:00 2024-04-25T18:29:11.140 INFO:teuthology.orchestra.run.smithi070.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-04-25T18:29:11.172 INFO:teuthology.orchestra.run.smithi114.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-25T18:29:11.239 INFO:teuthology.orchestra.run.smithi114.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-25T18:29:11.257 INFO:teuthology.orchestra.run.smithi070.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-04-25T18:29:11.298 INFO:teuthology.orchestra.run.smithi114.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-25T18:29:11.366 INFO:teuthology.orchestra.run.smithi070.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-04-25T18:29:11.380 INFO:teuthology.orchestra.run.smithi063.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3156.gd69f1 3.7 MB/s | 3.5 MB 00:00 2024-04-25T18:29:11.466 INFO:teuthology.orchestra.run.smithi070.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-04-25T18:29:11.554 INFO:teuthology.orchestra.run.smithi063.stdout:(32/152): ceph-mgr-diskprediction-local-19.0.0- 7.1 MB/s | 7.4 MB 00:01 2024-04-25T18:29:11.566 INFO:teuthology.orchestra.run.smithi070.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-04-25T18:29:11.587 INFO:teuthology.orchestra.run.smithi063.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3156.gd6 1.2 MB/s | 249 kB 00:00 2024-04-25T18:29:11.621 INFO:teuthology.orchestra.run.smithi063.stdout:(34/152): ceph-mgr-rook-19.0.0-3156.gd69f13d8.e 749 kB/s | 49 kB 00:00 2024-04-25T18:29:11.663 INFO:teuthology.orchestra.run.smithi063.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3156.gd 221 kB/s | 16 kB 00:00 2024-04-25T18:29:11.705 INFO:teuthology.orchestra.run.smithi063.stdout:(36/152): ceph-volume-19.0.0-3156.gd69f13d8.el7 3.2 MB/s | 273 kB 00:00 2024-04-25T18:29:11.748 INFO:teuthology.orchestra.run.smithi114.stdout:(42/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 5.6 MB/s | 49 MB 00:08 2024-04-25T18:29:11.790 INFO:teuthology.orchestra.run.smithi114.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 2.5 MB/s | 1.2 MB 00:00 2024-04-25T18:29:11.797 INFO:teuthology.orchestra.run.smithi063.stdout:(37/152): cephfs-top-19.0.0-3156.gd69f13d8.el7. 201 kB/s | 18 kB 00:00 2024-04-25T18:29:11.807 INFO:teuthology.orchestra.run.smithi114.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T18:29:11.832 INFO:teuthology.orchestra.run.smithi114.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T18:29:11.855 INFO:teuthology.orchestra.run.smithi063.stdout:(38/152): cephadm-19.0.0-3156.gd69f13d8.el7.noa 3.9 MB/s | 762 kB 00:00 2024-04-25T18:29:11.857 INFO:teuthology.orchestra.run.smithi114.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T18:29:11.883 INFO:teuthology.orchestra.run.smithi114.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T18:29:11.916 INFO:teuthology.orchestra.run.smithi114.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 12 MB/s | 689 kB 00:00 2024-04-25T18:29:11.958 INFO:teuthology.orchestra.run.smithi114.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-25T18:29:11.984 INFO:teuthology.orchestra.run.smithi114.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T18:29:12.009 INFO:teuthology.orchestra.run.smithi114.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-25T18:29:12.043 INFO:teuthology.orchestra.run.smithi114.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-25T18:29:12.047 INFO:teuthology.orchestra.run.smithi063.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 322 kB/s | 80 kB 00:00 2024-04-25T18:29:12.068 INFO:teuthology.orchestra.run.smithi114.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 565 kB/s | 33 kB 00:00 2024-04-25T18:29:12.114 INFO:teuthology.orchestra.run.smithi063.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-04-25T18:29:12.140 INFO:teuthology.orchestra.run.smithi063.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-25T18:29:12.160 INFO:teuthology.orchestra.run.smithi114.stdout:(54/152): ceph-mgr-diskprediction-local-19.0.0- 2.9 MB/s | 7.4 MB 00:02 2024-04-25T18:29:12.173 INFO:teuthology.orchestra.run.smithi063.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.3 MB/s | 253 kB 00:00 2024-04-25T18:29:12.218 INFO:teuthology.orchestra.run.smithi114.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-04-25T18:29:12.240 INFO:teuthology.orchestra.run.smithi063.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-25T18:29:12.243 INFO:teuthology.orchestra.run.smithi114.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 105 kB/s | 18 kB 00:00 2024-04-25T18:29:12.265 INFO:teuthology.orchestra.run.smithi063.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-04-25T18:29:12.269 INFO:teuthology.orchestra.run.smithi114.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 105 kB/s | 11 kB 00:00 2024-04-25T18:29:12.291 INFO:teuthology.orchestra.run.smithi063.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-04-25T18:29:12.294 INFO:teuthology.orchestra.run.smithi114.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T18:29:12.316 INFO:teuthology.orchestra.run.smithi063.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-25T18:29:12.319 INFO:teuthology.orchestra.run.smithi114.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-04-25T18:29:12.344 INFO:teuthology.orchestra.run.smithi114.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-25T18:29:12.350 INFO:teuthology.orchestra.run.smithi063.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 215 kB 00:00 2024-04-25T18:29:12.370 INFO:teuthology.orchestra.run.smithi114.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 582 kB 00:00 2024-04-25T18:29:12.392 INFO:teuthology.orchestra.run.smithi063.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 9.3 MB/s | 689 kB 00:00 2024-04-25T18:29:12.395 INFO:teuthology.orchestra.run.smithi114.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 343 kB/s | 25 kB 00:00 2024-04-25T18:29:12.420 INFO:teuthology.orchestra.run.smithi114.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 639 kB/s | 47 kB 00:00 2024-04-25T18:29:12.446 INFO:teuthology.orchestra.run.smithi114.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T18:29:12.458 INFO:teuthology.orchestra.run.smithi063.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 9.2 MB/s | 943 kB 00:00 2024-04-25T18:29:12.470 INFO:teuthology.orchestra.run.smithi114.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 188 kB 00:00 2024-04-25T18:29:12.484 INFO:teuthology.orchestra.run.smithi063.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 107 kB 00:00 2024-04-25T18:29:12.496 INFO:teuthology.orchestra.run.smithi114.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 473 kB/s | 35 kB 00:00 2024-04-25T18:29:12.509 INFO:teuthology.orchestra.run.smithi063.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 445 kB/s | 21 kB 00:00 2024-04-25T18:29:12.543 INFO:teuthology.orchestra.run.smithi063.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-25T18:29:12.568 INFO:teuthology.orchestra.run.smithi063.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 587 kB/s | 33 kB 00:00 2024-04-25T18:29:12.621 INFO:teuthology.orchestra.run.smithi114.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-04-25T18:29:12.635 INFO:teuthology.orchestra.run.smithi063.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 303 kB/s | 18 kB 00:00 2024-04-25T18:29:12.688 INFO:teuthology.orchestra.run.smithi114.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-25T18:29:12.727 INFO:teuthology.orchestra.run.smithi063.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-25T18:29:12.730 INFO:teuthology.orchestra.run.smithi114.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 6.9 MB/s | 1.6 MB 00:00 2024-04-25T18:29:12.744 INFO:teuthology.orchestra.run.smithi063.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 116 kB/s | 11 kB 00:00 2024-04-25T18:29:12.755 INFO:teuthology.orchestra.run.smithi114.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 6.9 MB/s | 911 kB 00:00 2024-04-25T18:29:12.780 INFO:teuthology.orchestra.run.smithi114.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-25T18:29:12.802 INFO:teuthology.orchestra.run.smithi063.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-25T18:29:12.820 INFO:teuthology.orchestra.run.smithi063.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 619 kB/s | 45 kB 00:00 2024-04-25T18:29:12.822 INFO:teuthology.orchestra.run.smithi114.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-25T18:29:12.847 INFO:teuthology.orchestra.run.smithi114.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-25T18:29:12.870 INFO:teuthology.orchestra.run.smithi063.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 662 kB 00:00 2024-04-25T18:29:12.912 INFO:teuthology.orchestra.run.smithi063.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-04-25T18:29:12.916 INFO:teuthology.orchestra.run.smithi070.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 774 kB/s | 1.0 MB 00:01 2024-04-25T18:29:12.937 INFO:teuthology.orchestra.run.smithi063.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 420 kB/s | 25 kB 00:00 2024-04-25T18:29:12.962 INFO:teuthology.orchestra.run.smithi063.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 968 kB/s | 47 kB 00:00 2024-04-25T18:29:12.996 INFO:teuthology.orchestra.run.smithi063.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.0 MB/s | 292 kB 00:00 2024-04-25T18:29:12.998 INFO:teuthology.orchestra.run.smithi114.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 6.0 MB 00:00 2024-04-25T18:29:13.017 INFO:teuthology.orchestra.run.smithi070.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-25T18:29:13.023 INFO:teuthology.orchestra.run.smithi114.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-04-25T18:29:13.029 INFO:teuthology.orchestra.run.smithi063.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 188 kB 00:00 2024-04-25T18:29:13.057 INFO:teuthology.orchestra.run.smithi063.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 607 kB/s | 35 kB 00:00 2024-04-25T18:29:13.107 INFO:teuthology.orchestra.run.smithi114.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-04-25T18:29:13.132 INFO:teuthology.orchestra.run.smithi114.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-25T18:29:13.142 INFO:teuthology.orchestra.run.smithi070.stdout:(130/152): python3-grpcio-1.46.7-10.el9.x86_64. 747 kB/s | 2.0 MB 00:02 2024-04-25T18:29:13.157 INFO:teuthology.orchestra.run.smithi114.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 193 kB/s | 26 kB 00:00 2024-04-25T18:29:13.182 INFO:teuthology.orchestra.run.smithi114.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-25T18:29:13.192 INFO:teuthology.orchestra.run.smithi070.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 450 kB/s | 79 kB 00:00 2024-04-25T18:29:13.207 INFO:teuthology.orchestra.run.smithi063.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-04-25T18:29:13.208 INFO:teuthology.orchestra.run.smithi114.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T18:29:13.233 INFO:teuthology.orchestra.run.smithi114.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-04-25T18:29:13.258 INFO:teuthology.orchestra.run.smithi114.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-25T18:29:13.283 INFO:teuthology.orchestra.run.smithi114.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-25T18:29:13.309 INFO:teuthology.orchestra.run.smithi114.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 77 kB 00:00 2024-04-25T18:29:13.324 INFO:teuthology.orchestra.run.smithi063.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-25T18:29:13.326 INFO:teuthology.orchestra.run.smithi070.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 314 kB/s | 58 kB 00:00 2024-04-25T18:29:13.334 INFO:teuthology.orchestra.run.smithi114.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 886 kB/s | 42 kB 00:00 2024-04-25T18:29:13.374 INFO:teuthology.orchestra.run.smithi063.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-25T18:29:13.401 INFO:teuthology.orchestra.run.smithi114.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 5.1 MB 00:00 2024-04-25T18:29:13.416 INFO:teuthology.orchestra.run.smithi063.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T18:29:13.426 INFO:teuthology.orchestra.run.smithi114.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-25T18:29:13.434 INFO:teuthology.orchestra.run.smithi070.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-25T18:29:13.441 INFO:teuthology.orchestra.run.smithi063.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T18:29:13.451 INFO:teuthology.orchestra.run.smithi114.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-25T18:29:13.476 INFO:teuthology.orchestra.run.smithi114.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T18:29:13.493 INFO:teuthology.orchestra.run.smithi114.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 764 kB/s | 51 kB 00:00 2024-04-25T18:29:13.518 INFO:teuthology.orchestra.run.smithi114.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 809 kB/s | 54 kB 00:00 2024-04-25T18:29:13.550 INFO:teuthology.orchestra.run.smithi063.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-25T18:29:13.594 INFO:teuthology.orchestra.run.smithi114.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 417 kB/s | 42 kB 00:00 2024-04-25T18:29:13.635 INFO:teuthology.orchestra.run.smithi070.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 616 kB/s | 272 kB 00:00 2024-04-25T18:29:13.708 INFO:teuthology.orchestra.run.smithi114.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 314 kB/s | 44 kB 00:00 2024-04-25T18:29:13.708 INFO:teuthology.orchestra.run.smithi070.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 399 kB/s | 90 kB 00:00 2024-04-25T18:29:13.708 INFO:teuthology.orchestra.run.smithi063.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-04-25T18:29:13.708 INFO:teuthology.orchestra.run.smithi063.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 685 kB/s | 87 kB 00:00 2024-04-25T18:29:13.727 INFO:teuthology.orchestra.run.smithi070.stdout:(136/152): libarrow-9.0.0-10.el9.x86_64.rpm 766 kB/s | 4.4 MB 00:05 2024-04-25T18:29:13.751 INFO:teuthology.orchestra.run.smithi063.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 3.9 MB/s | 244 kB 00:00 2024-04-25T18:29:13.752 INFO:teuthology.orchestra.run.smithi070.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-04-25T18:29:13.778 INFO:teuthology.orchestra.run.smithi070.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 262 kB/s | 31 kB 00:00 2024-04-25T18:29:13.853 INFO:teuthology.orchestra.run.smithi114.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-25T18:29:13.878 INFO:teuthology.orchestra.run.smithi114.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 785 kB/s | 221 kB 00:00 2024-04-25T18:29:13.895 INFO:teuthology.orchestra.run.smithi114.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 155 kB/s | 36 kB 00:00 2024-04-25T18:29:13.901 INFO:teuthology.orchestra.run.smithi063.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-04-25T18:29:13.903 INFO:teuthology.orchestra.run.smithi070.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 286 kB/s | 36 kB 00:00 2024-04-25T18:29:13.920 INFO:teuthology.orchestra.run.smithi114.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 714 kB/s | 47 kB 00:00 2024-04-25T18:29:13.935 INFO:teuthology.orchestra.run.smithi063.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-25T18:29:13.945 INFO:teuthology.orchestra.run.smithi114.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-25T18:29:13.971 INFO:teuthology.orchestra.run.smithi114.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 768 kB/s | 58 kB 00:00 2024-04-25T18:29:13.977 INFO:teuthology.orchestra.run.smithi063.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 636 kB/s | 26 kB 00:00 2024-04-25T18:29:13.996 INFO:teuthology.orchestra.run.smithi114.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T18:29:14.002 INFO:teuthology.orchestra.run.smithi063.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 294 kB/s | 19 kB 00:00 2024-04-25T18:29:14.020 INFO:teuthology.orchestra.run.smithi070.stdout:(140/152): python3-typing-extensions-4.4.0-2.el 437 kB/s | 51 kB 00:00 2024-04-25T18:29:14.021 INFO:teuthology.orchestra.run.smithi114.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-04-25T18:29:14.045 INFO:teuthology.orchestra.run.smithi070.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 594 kB/s | 188 kB 00:00 2024-04-25T18:29:14.046 INFO:teuthology.orchestra.run.smithi114.stdout:(102/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-04-25T18:29:14.052 INFO:teuthology.orchestra.run.smithi063.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 716 kB/s | 35 kB 00:00 2024-04-25T18:29:14.071 INFO:teuthology.orchestra.run.smithi114.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-25T18:29:14.085 INFO:teuthology.orchestra.run.smithi063.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-25T18:29:14.113 INFO:teuthology.orchestra.run.smithi114.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.2 MB/s | 863 kB 00:00 2024-04-25T18:29:14.188 INFO:teuthology.orchestra.run.smithi114.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 260 kB/s | 19 kB 00:00 2024-04-25T18:29:14.237 INFO:teuthology.orchestra.run.smithi070.stdout:(142/152): python3-websocket-client-1.2.3-2.el9 468 kB/s | 90 kB 00:00 2024-04-25T18:29:14.239 INFO:teuthology.orchestra.run.smithi114.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-04-25T18:29:14.264 INFO:teuthology.orchestra.run.smithi114.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-25T18:29:14.279 INFO:teuthology.orchestra.run.smithi070.stdout:(143/152): python3-rsa-4.9-2.el9.noarch.rpm 112 kB/s | 59 kB 00:00 2024-04-25T18:29:14.289 INFO:teuthology.orchestra.run.smithi114.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 517 kB/s | 26 kB 00:00 2024-04-25T18:29:14.315 INFO:teuthology.orchestra.run.smithi114.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-04-25T18:29:14.406 INFO:teuthology.orchestra.run.smithi070.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 158 kB/s | 20 kB 00:00 2024-04-25T18:29:14.429 INFO:teuthology.orchestra.run.smithi070.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 561 kB/s | 230 kB 00:00 2024-04-25T18:29:14.441 INFO:teuthology.orchestra.run.smithi114.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 444 kB/s | 67 kB 00:00 2024-04-25T18:29:14.591 INFO:teuthology.orchestra.run.smithi114.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 548 kB/s | 151 kB 00:00 2024-04-25T18:29:14.625 INFO:teuthology.orchestra.run.smithi114.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-04-25T18:29:14.658 INFO:teuthology.orchestra.run.smithi114.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T18:29:14.683 INFO:teuthology.orchestra.run.smithi114.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-04-25T18:29:14.709 INFO:teuthology.orchestra.run.smithi114.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-04-25T18:29:14.734 INFO:teuthology.orchestra.run.smithi114.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T18:29:14.736 INFO:teuthology.orchestra.run.smithi063.stdout:(81/152): ceph-test-19.0.0-3156.gd69f13d8.el7.x 7.3 MB/s | 49 MB 00:06 2024-04-25T18:29:14.755 INFO:teuthology.orchestra.run.smithi070.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 549 kB/s | 192 kB 00:00 2024-04-25T18:29:14.759 INFO:teuthology.orchestra.run.smithi114.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-25T18:29:14.784 INFO:teuthology.orchestra.run.smithi114.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-25T18:29:14.786 INFO:teuthology.orchestra.run.smithi063.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 359 kB/s | 250 kB 00:00 2024-04-25T18:29:14.818 INFO:teuthology.orchestra.run.smithi114.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-04-25T18:29:14.830 INFO:teuthology.orchestra.run.smithi070.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 721 kB/s | 427 kB 00:00 2024-04-25T18:29:14.843 INFO:teuthology.orchestra.run.smithi114.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-04-25T18:29:14.868 INFO:teuthology.orchestra.run.smithi114.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T18:29:14.920 INFO:teuthology.orchestra.run.smithi063.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.9 MB/s | 5.1 MB 00:00 2024-04-25T18:29:14.935 INFO:teuthology.orchestra.run.smithi114.stdout:(122/152): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-04-25T18:29:14.945 INFO:teuthology.orchestra.run.smithi063.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 222 kB 00:00 2024-04-25T18:29:14.961 INFO:teuthology.orchestra.run.smithi114.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-25T18:29:14.970 INFO:teuthology.orchestra.run.smithi063.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 457 kB/s | 77 kB 00:00 2024-04-25T18:29:14.986 INFO:teuthology.orchestra.run.smithi114.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-04-25T18:29:15.011 INFO:teuthology.orchestra.run.smithi114.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-04-25T18:29:15.029 INFO:teuthology.orchestra.run.smithi063.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 742 kB/s | 42 kB 00:00 2024-04-25T18:29:15.036 INFO:teuthology.orchestra.run.smithi114.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-25T18:29:15.062 INFO:teuthology.orchestra.run.smithi114.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-25T18:29:15.062 INFO:teuthology.orchestra.run.smithi063.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-04-25T18:29:15.087 INFO:teuthology.orchestra.run.smithi114.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-25T18:29:15.105 INFO:teuthology.orchestra.run.smithi063.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-04-25T18:29:15.130 INFO:teuthology.orchestra.run.smithi063.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.8 MB/s | 279 kB 00:00 2024-04-25T18:29:15.130 INFO:teuthology.orchestra.run.smithi070.stdout:(148/152): librados2-19.0.0-3156.gd69f13d8.el7. 9.1 MB/s | 3.4 MB 00:00 2024-04-25T18:29:15.137 INFO:teuthology.orchestra.run.smithi114.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.9 MB/s | 2.0 MB 00:00 2024-04-25T18:29:15.155 INFO:teuthology.orchestra.run.smithi063.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 588 kB/s | 51 kB 00:00 2024-04-25T18:29:15.162 INFO:teuthology.orchestra.run.smithi114.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-04-25T18:29:15.188 INFO:teuthology.orchestra.run.smithi114.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T18:29:15.189 INFO:teuthology.orchestra.run.smithi063.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 937 kB/s | 54 kB 00:00 2024-04-25T18:29:15.205 INFO:teuthology.orchestra.run.smithi070.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 18 MB/s | 1.3 MB 00:00 2024-04-25T18:29:15.221 INFO:teuthology.orchestra.run.smithi114.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 989 kB/s | 58 kB 00:00 2024-04-25T18:29:15.255 INFO:teuthology.orchestra.run.smithi114.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-04-25T18:29:15.264 INFO:teuthology.orchestra.run.smithi070.stdout:(150/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 7.2 MB/s | 3.1 MB 00:00 2024-04-25T18:29:15.272 INFO:teuthology.orchestra.run.smithi063.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 359 kB/s | 42 kB 00:00 2024-04-25T18:29:15.280 INFO:teuthology.orchestra.run.smithi114.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-25T18:29:15.305 INFO:teuthology.orchestra.run.smithi114.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-25T18:29:15.331 INFO:teuthology.orchestra.run.smithi114.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-25T18:29:15.348 INFO:teuthology.orchestra.run.smithi114.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 733 kB/s | 31 kB 00:00 2024-04-25T18:29:15.356 INFO:teuthology.orchestra.run.smithi070.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-04-25T18:29:15.373 INFO:teuthology.orchestra.run.smithi114.stdout:(138/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 764 kB/s | 90 kB 00:00 2024-04-25T18:29:15.389 INFO:teuthology.orchestra.run.smithi063.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 229 kB/s | 44 kB 00:00 2024-04-25T18:29:15.390 INFO:teuthology.orchestra.run.smithi114.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-25T18:29:15.407 INFO:teuthology.orchestra.run.smithi114.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-04-25T18:29:15.432 INFO:teuthology.orchestra.run.smithi114.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T18:29:15.449 INFO:teuthology.orchestra.run.smithi114.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 864 kB/s | 51 kB 00:00 2024-04-25T18:29:15.474 INFO:teuthology.orchestra.run.smithi114.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-25T18:29:15.500 INFO:teuthology.orchestra.run.smithi114.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T18:29:15.523 INFO:teuthology.orchestra.run.smithi063.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 918 kB/s | 221 kB 00:00 2024-04-25T18:29:15.525 INFO:teuthology.orchestra.run.smithi114.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-04-25T18:29:15.550 INFO:teuthology.orchestra.run.smithi114.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-25T18:29:15.576 INFO:teuthology.orchestra.run.smithi114.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T18:29:15.618 INFO:teuthology.orchestra.run.smithi114.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-25T18:29:15.676 INFO:teuthology.orchestra.run.smithi114.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T18:29:15.759 INFO:teuthology.orchestra.run.smithi114.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T18:29:15.840 INFO:teuthology.orchestra.run.smithi063.stdout:(95/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-25T18:29:15.865 INFO:teuthology.orchestra.run.smithi063.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 78 kB/s | 36 kB 00:00 2024-04-25T18:29:15.890 INFO:teuthology.orchestra.run.smithi063.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-04-25T18:29:15.932 INFO:teuthology.orchestra.run.smithi063.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 7.4 MB/s | 305 kB 00:00 2024-04-25T18:29:15.999 INFO:teuthology.orchestra.run.smithi063.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 541 kB/s | 58 kB 00:00 2024-04-25T18:29:16.200 INFO:teuthology.orchestra.run.smithi063.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 305 kB/s | 94 kB 00:00 2024-04-25T18:29:16.225 INFO:teuthology.orchestra.run.smithi063.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 219 kB/s | 64 kB 00:00 2024-04-25T18:29:16.243 INFO:teuthology.orchestra.run.smithi114.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 4.7 MB/s | 3.1 MB 00:00 2024-04-25T18:29:16.250 INFO:teuthology.orchestra.run.smithi063.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 926 kB/s | 45 kB 00:00 2024-04-25T18:29:16.275 INFO:teuthology.orchestra.run.smithi063.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 304 kB/s | 22 kB 00:00 2024-04-25T18:29:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.3 MB/s | 863 kB 00:00 2024-04-25T18:29:16.393 INFO:teuthology.orchestra.run.smithi063.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 262 kB/s | 19 kB 00:00 2024-04-25T18:29:16.443 INFO:teuthology.orchestra.run.smithi114.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 3.8 MB/s | 3.4 MB 00:00 2024-04-25T18:29:16.447 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-04-25T18:29:16.447 INFO:teuthology.orchestra.run.smithi114.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-25T18:29:16.472 INFO:teuthology.orchestra.run.smithi070.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 793 kB/s | 1.6 MB 00:02 2024-04-25T18:29:16.479 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-25T18:29:16.480 INFO:teuthology.orchestra.run.smithi070.stdout:Total 10 MB/s | 222 MB 00:21 2024-04-25T18:29:16.492 INFO:teuthology.orchestra.run.smithi063.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-04-25T18:29:16.576 INFO:teuthology.orchestra.run.smithi063.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-04-25T18:29:16.601 INFO:teuthology.orchestra.run.smithi063.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-04-25T18:29:16.627 INFO:teuthology.orchestra.run.smithi063.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 985 kB/s | 49 kB 00:00 2024-04-25T18:29:16.652 INFO:teuthology.orchestra.run.smithi063.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T18:29:16.686 INFO:teuthology.orchestra.run.smithi063.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.6 MB/s | 151 kB 00:00 2024-04-25T18:29:16.861 INFO:teuthology.orchestra.run.smithi063.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-04-25T18:29:16.903 INFO:teuthology.orchestra.run.smithi063.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-04-25T18:29:16.928 INFO:teuthology.orchestra.run.smithi063.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 658 kB/s | 43 kB 00:00 2024-04-25T18:29:16.953 INFO:teuthology.orchestra.run.smithi063.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 650 kB/s | 32 kB 00:00 2024-04-25T18:29:16.979 INFO:teuthology.orchestra.run.smithi063.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 304 kB/s | 15 kB 00:00 2024-04-25T18:29:17.012 INFO:teuthology.orchestra.run.smithi063.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T18:29:17.062 INFO:teuthology.orchestra.run.smithi063.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T18:29:17.096 INFO:teuthology.orchestra.run.smithi063.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-25T18:29:17.121 INFO:teuthology.orchestra.run.smithi063.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 612 kB/s | 35 kB 00:00 2024-04-25T18:29:17.272 INFO:teuthology.orchestra.run.smithi063.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.2 MB/s | 206 kB 00:00 2024-04-25T18:29:17.380 INFO:teuthology.orchestra.run.smithi063.stdout:(122/152): libarrow-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2024-04-25T18:29:17.414 INFO:teuthology.orchestra.run.smithi063.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-25T18:29:17.439 INFO:teuthology.orchestra.run.smithi063.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-04-25T18:29:17.464 INFO:teuthology.orchestra.run.smithi063.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 358 kB/s | 18 kB 00:00 2024-04-25T18:29:17.528 INFO:teuthology.orchestra.run.smithi063.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 265 kB/s | 23 kB 00:00 2024-04-25T18:29:17.536 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-04-25T18:29:17.557 INFO:teuthology.orchestra.run.smithi063.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 213 kB/s | 19 kB 00:00 2024-04-25T18:29:17.573 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-25T18:29:17.582 INFO:teuthology.orchestra.run.smithi063.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 368 kB/s | 20 kB 00:00 2024-04-25T18:29:17.649 INFO:teuthology.orchestra.run.smithi063.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.9 MB/s | 2.0 MB 00:00 2024-04-25T18:29:17.674 INFO:teuthology.orchestra.run.smithi063.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 507 kB/s | 46 kB 00:00 2024-04-25T18:29:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-04-25T18:29:17.678 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-04-25T18:29:17.699 INFO:teuthology.orchestra.run.smithi063.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-04-25T18:29:17.712 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-25T18:29:17.713 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-25T18:29:17.725 INFO:teuthology.orchestra.run.smithi063.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-25T18:29:17.775 INFO:teuthology.orchestra.run.smithi063.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-25T18:29:17.801 INFO:teuthology.orchestra.run.smithi063.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 222 kB/s | 16 kB 00:00 2024-04-25T18:29:17.876 INFO:teuthology.orchestra.run.smithi063.stdout:(135/152): python3-kubernetes-26.1.0-2.el9.noar 3.2 MB/s | 1.0 MB 00:00 2024-04-25T18:29:17.901 INFO:teuthology.orchestra.run.smithi063.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 284 kB/s | 28 kB 00:00 2024-04-25T18:29:17.926 INFO:teuthology.orchestra.run.smithi063.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 623 kB/s | 31 kB 00:00 2024-04-25T18:29:17.951 INFO:teuthology.orchestra.run.smithi063.stdout:(138/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 513 kB/s | 90 kB 00:00 2024-04-25T18:29:17.985 INFO:teuthology.orchestra.run.smithi063.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-25T18:29:18.010 INFO:teuthology.orchestra.run.smithi063.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 621 kB/s | 36 kB 00:00 2024-04-25T18:29:18.036 INFO:teuthology.orchestra.run.smithi063.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T18:29:18.061 INFO:teuthology.orchestra.run.smithi063.stdout:(142/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-25T18:29:18.086 INFO:teuthology.orchestra.run.smithi063.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-25T18:29:18.119 INFO:teuthology.orchestra.run.smithi063.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T18:29:18.145 INFO:teuthology.orchestra.run.smithi063.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-04-25T18:29:18.178 INFO:teuthology.orchestra.run.smithi063.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T18:29:18.237 INFO:teuthology.orchestra.run.smithi063.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-25T18:29:18.404 INFO:teuthology.orchestra.run.smithi063.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-04-25T18:29:18.488 INFO:teuthology.orchestra.run.smithi063.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-04-25T18:29:18.621 INFO:teuthology.orchestra.run.smithi063.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-25T18:29:18.780 INFO:teuthology.orchestra.run.smithi063.stdout:(151/152): librbd1-19.0.0-3156.gd69f13d8.el7.x8 5.8 MB/s | 3.1 MB 00:00 2024-04-25T18:29:18.914 INFO:teuthology.orchestra.run.smithi063.stdout:(152/152): librados2-19.0.0-3156.gd69f13d8.el7. 4.6 MB/s | 3.4 MB 00:00 2024-04-25T18:29:18.924 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-25T18:29:18.925 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 222 MB 00:17 2024-04-25T18:29:19.429 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-04-25T18:29:19.429 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-04-25T18:29:19.479 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-25T18:29:19.480 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-25T18:29:21.930 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-25T18:29:22.192 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-25T18:29:22.192 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-25T18:29:24.883 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-25T18:29:24.884 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-25T18:29:25.480 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-04-25T18:29:25.563 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T18:29:25.614 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-25T18:29:25.685 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T18:29:25.709 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T18:29:25.755 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T18:29:25.810 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T18:29:25.843 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T18:29:25.868 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T18:29:25.907 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T18:29:25.911 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T18:29:25.957 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T18:29:25.982 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T18:29:26.026 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T18:29:26.250 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T18:29:26.474 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:26.481 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:26.529 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:26.595 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:26.603 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:26.625 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:26.633 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:29:26.717 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:26.755 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:29:26.780 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T18:29:26.838 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T18:29:26.866 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:26.902 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T18:29:26.951 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T18:29:26.969 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:26.979 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:26.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:27.081 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:27.105 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:27.161 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:27.183 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:27.204 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:27.309 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:27.342 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:27.399 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:27.433 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:27.450 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T18:29:27.539 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T18:29:27.545 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T18:29:27.575 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T18:29:27.610 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T18:29:27.634 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T18:29:27.665 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T18:29:27.674 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T18:29:27.699 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T18:29:27.734 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T18:29:27.763 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T18:29:27.778 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T18:29:27.798 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T18:29:27.810 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T18:29:27.838 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T18:29:27.842 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T18:29:27.874 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T18:29:27.903 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T18:29:27.908 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T18:29:27.961 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T18:29:27.971 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T18:29:27.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T18:29:28.025 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T18:29:28.036 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T18:29:28.056 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T18:29:28.096 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T18:29:28.113 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T18:29:28.154 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T18:29:28.166 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T18:29:28.198 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T18:29:28.202 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T18:29:28.226 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T18:29:28.245 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T18:29:28.274 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T18:29:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T18:29:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T18:29:28.343 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T18:29:28.388 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T18:29:28.416 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T18:29:28.462 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T18:29:28.550 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T18:29:28.593 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T18:29:28.602 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T18:29:28.641 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T18:29:29.131 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T18:29:29.156 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T18:29:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T18:29:29.236 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T18:29:29.265 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T18:29:29.293 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T18:29:29.308 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T18:29:29.346 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T18:29:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T18:29:29.387 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T18:29:29.407 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T18:29:29.449 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T18:29:29.541 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T18:29:29.595 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T18:29:29.611 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T18:29:29.651 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T18:29:29.670 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T18:29:29.706 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T18:29:29.752 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T18:29:29.779 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T18:29:29.810 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T18:29:29.819 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T18:29:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T18:29:29.866 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T18:29:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T18:29:29.910 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T18:29:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T18:29:29.953 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T18:29:29.973 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T18:29:29.995 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T18:29:30.015 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T18:29:30.138 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T18:29:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T18:29:30.184 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T18:29:30.203 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T18:29:30.234 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T18:29:30.248 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T18:29:30.275 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T18:29:30.369 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T18:29:30.400 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T18:29:30.425 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T18:29:30.432 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T18:29:30.466 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T18:29:30.491 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T18:29:30.565 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T18:29:30.599 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T18:29:30.631 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T18:29:30.662 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T18:29:30.688 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T18:29:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T18:29:30.713 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T18:29:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T18:29:30.759 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T18:29:30.788 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T18:29:30.793 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T18:29:30.833 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T18:29:30.864 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T18:29:30.894 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T18:29:30.913 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T18:29:30.929 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T18:29:30.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T18:29:30.954 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T18:29:30.973 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T18:29:30.995 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T18:29:31.007 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T18:29:31.030 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T18:29:31.046 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T18:29:31.078 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T18:29:31.108 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T18:29:31.116 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T18:29:31.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T18:29:31.159 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T18:29:31.160 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T18:29:31.191 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T18:29:31.200 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T18:29:31.235 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T18:29:31.238 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T18:29:31.321 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T18:29:31.336 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T18:29:31.365 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T18:29:31.393 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T18:29:31.397 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T18:29:31.443 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T18:29:31.541 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T18:29:31.598 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T18:29:31.692 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-25T18:29:31.862 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T18:29:31.940 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T18:29:32.058 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T18:29:32.111 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T18:29:32.127 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T18:29:32.157 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T18:29:32.189 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T18:29:32.190 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T18:29:32.288 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T18:29:32.314 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T18:29:32.327 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T18:29:32.361 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T18:29:32.395 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T18:29:32.423 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T18:29:32.492 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T18:29:32.510 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T18:29:32.544 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T18:29:32.559 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T18:29:32.638 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T18:29:32.672 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T18:29:32.737 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T18:29:32.765 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T18:29:32.791 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T18:29:32.843 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T18:29:32.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T18:29:32.878 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T18:29:32.943 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:32.955 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T18:29:33.122 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:29:33.154 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:33.296 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:29:33.348 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:29:33.484 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T18:29:33.594 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T18:29:33.603 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T18:29:33.700 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T18:29:33.815 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T18:29:33.893 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T18:29:33.925 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:34.126 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:29:34.159 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:34.262 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 14/156 2024-04-25T18:29:34.293 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:34.368 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T18:29:34.659 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:34.719 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T18:29:34.812 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T18:29:34.885 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T18:29:34.906 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T18:29:34.913 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T18:29:34.922 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T18:29:34.946 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T18:29:34.982 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T18:29:35.010 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T18:29:35.043 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T18:29:35.046 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T18:29:35.102 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T18:29:35.153 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T18:29:35.256 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T18:29:35.260 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T18:29:35.293 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:35.306 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T18:29:35.353 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T18:29:35.382 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:35.398 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:35.398 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T18:29:35.414 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T18:29:35.450 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T18:29:35.453 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T18:29:35.470 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:35.493 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T18:29:35.502 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T18:29:35.538 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T18:29:35.617 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T18:29:35.702 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T18:29:35.753 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T18:29:35.817 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T18:29:35.925 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T18:29:35.936 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T18:29:35.954 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:35.980 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 31/156 2024-04-25T18:29:36.015 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T18:29:36.024 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:36.029 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 32/156 2024-04-25T18:29:36.047 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T18:29:36.050 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:36.079 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 33/156 2024-04-25T18:29:36.121 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:36.143 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T18:29:36.202 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T18:29:36.260 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T18:29:36.383 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T18:29:36.574 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T18:29:36.645 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T18:29:37.361 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T18:29:37.488 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T18:29:37.527 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T18:29:37.572 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T18:29:37.613 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 43/156 2024-04-25T18:29:37.689 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T18:29:38.012 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T18:29:38.073 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T18:29:38.184 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T18:29:38.285 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T18:29:38.322 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.329 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T18:29:38.351 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.369 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T18:29:38.396 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.431 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.447 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T18:29:38.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T18:29:38.589 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T18:29:38.662 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T18:29:38.872 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T18:29:38.891 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.931 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T18:29:38.938 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:38.971 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:38.971 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T18:29:39.010 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:39.016 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:39.016 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T18:29:39.016 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T18:29:39.016 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:39.029 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T18:29:39.048 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:29:39.101 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:39.101 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T18:29:39.102 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T18:29:39.102 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:39.128 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:29:39.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T18:29:39.228 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T18:29:39.280 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T18:29:39.318 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T18:29:39.351 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T18:29:39.549 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T18:29:39.728 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T18:29:39.782 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T18:29:39.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T18:29:39.896 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T18:29:39.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T18:29:40.032 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T18:29:40.079 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T18:29:40.133 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T18:29:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T18:29:40.277 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T18:29:40.323 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T18:29:40.462 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T18:29:40.543 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T18:29:40.604 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T18:29:40.678 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T18:29:40.836 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T18:29:40.933 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T18:29:41.948 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T18:29:42.008 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T18:29:42.041 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T18:29:42.169 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T18:29:42.236 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T18:29:42.373 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T18:29:42.416 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T18:29:42.663 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T18:29:42.697 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T18:29:42.797 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T18:29:43.656 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T18:29:43.816 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T18:29:45.494 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T18:29:45.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T18:29:45.651 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T18:29:45.699 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T18:29:46.101 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T18:29:46.164 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:46.291 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 99/156 2024-04-25T18:29:46.322 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 100/156 2024-04-25T18:29:46.381 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T18:29:46.955 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T18:29:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:47.205 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 103/156 2024-04-25T18:29:47.245 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 104/156 2024-04-25T18:29:50.160 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:50.199 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-04-25T18:29:50.460 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-04-25T18:29:50.461 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-04-25T18:29:50.461 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:50.529 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-04-25T18:29:50.530 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:50.631 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:29:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:29:50.650 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:50.650 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T18:29:50.650 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:50.700 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:29:50.716 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:29:50.716 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:50.716 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T18:29:50.716 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:50.896 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 105/156 2024-04-25T18:29:51.110 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:51.335 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 106/156 2024-04-25T18:29:51.335 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T18:29:51.335 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T18:29:51.336 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:29:51.377 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:29:52.134 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:29:52.160 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:52.203 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:29:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:29:52.229 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.229 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T18:29:52.229 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:29:52.229 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:29:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:52.385 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:29:52.408 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:52.445 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:29:52.467 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:29:52.467 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.468 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T18:29:52.468 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:29:52.468 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:29:52.468 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:52.844 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:29:52.866 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:29:52.866 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.866 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T18:29:52.866 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:29:52.866 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:29:52.867 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:52.904 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:29:52.926 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:29:52.927 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:52.927 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T18:29:52.927 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:29:52.927 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:29:52.927 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:52.966 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T18:29:53.015 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T18:29:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T18:29:53.069 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T18:29:53.108 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T18:29:53.163 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T18:29:53.252 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T18:29:53.303 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T18:29:53.347 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T18:29:53.380 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T18:29:53.388 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T18:29:53.466 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T18:29:53.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:29:53.562 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:29:53.608 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:29:53.645 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T18:29:53.676 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T18:29:53.701 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:29:53.704 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T18:29:53.738 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T18:29:53.770 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T18:29:53.798 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T18:29:54.130 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:29:54.161 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:29:54.193 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T18:29:54.250 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:29:54.254 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T18:29:54.279 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:29:54.284 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:29:54.312 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T18:29:54.373 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T18:29:54.411 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:29:55.316 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:29:55.361 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:29:55.447 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:29:55.480 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:29:56.320 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:29:56.353 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:29:56.441 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:29:56.454 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:29:56.463 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:29:56.565 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:29:56.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T18:29:56.646 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:29:56.669 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:56.714 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:29:56.718 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T18:29:56.725 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:29:56.748 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:29:56.772 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:56.816 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:29:56.822 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T18:29:56.828 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:29:56.871 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T18:29:56.924 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T18:29:56.972 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T18:29:57.790 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T18:29:57.833 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:29:57.855 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:29:57.855 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:57.855 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T18:29:57.856 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:29:57.856 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:29:57.856 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:57.893 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T18:29:57.900 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:29:57.918 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:29:57.922 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:29:57.922 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:57.922 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T18:29:57.922 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:57.940 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:29:57.941 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:57.941 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T18:29:57.941 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:29:57.941 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:29:57.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:57.967 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:29:57.985 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:29:57.989 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:29:57.989 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:57.989 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T18:29:57.990 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:29:57.990 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:29:57.990 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:58.007 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:29:58.007 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:58.007 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T18:29:58.007 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:58.053 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:29:58.075 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:29:58.293 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:29:58.315 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:29:58.315 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:58.315 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T18:29:58.315 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:29:58.315 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:29:58.316 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:29:58.387 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:29:58.409 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:29:58.409 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:29:58.409 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T18:29:58.410 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:29:58.410 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:29:58.410 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:30:02.897 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T18:30:02.938 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T18:30:02.979 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T18:30:02.981 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T18:30:03.011 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T18:30:03.022 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T18:30:03.039 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T18:30:03.063 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T18:30:03.095 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T18:30:03.122 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T18:30:03.211 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T18:30:03.251 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T18:30:03.281 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T18:30:03.311 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T18:30:03.352 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T18:30:03.382 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T18:30:03.851 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:03.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:03.907 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:03.977 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:04.013 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:04.033 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:04.732 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T18:30:04.825 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T18:30:04.854 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T18:30:04.877 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T18:30:04.905 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 107/156 2024-04-25T18:30:04.905 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-04-25T18:30:04.905 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-04-25T18:30:04.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:04.947 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T18:30:04.951 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T18:30:04.987 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T18:30:04.988 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:04.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T18:30:05.071 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:05.073 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T18:30:05.104 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:05.104 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:05.105 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T18:30:05.105 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:05.123 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:30:05.175 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 108/156 2024-04-25T18:30:05.175 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:05.175 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T18:30:05.175 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:05.189 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:05.196 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:05.231 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:05.231 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:05.262 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:05.317 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:05.387 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:06.833 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:30:06.915 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 109/156 2024-04-25T18:30:06.915 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:06.915 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T18:30:06.916 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:30:06.916 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T18:30:06.916 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:07.175 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:30:07.251 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 110/156 2024-04-25T18:30:07.252 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:07.252 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T18:30:07.252 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:30:07.252 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T18:30:07.252 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:07.800 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:30:07.872 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 111/156 2024-04-25T18:30:07.872 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:07.872 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T18:30:07.872 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:30:07.873 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T18:30:07.873 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:08.027 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T18:30:08.089 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T18:30:08.166 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T18:30:08.547 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T18:30:08.615 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T18:30:08.708 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T18:30:08.832 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:30:09.010 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:30:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T18:30:09.147 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T18:30:09.178 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T18:30:09.901 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:30:09.949 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 123/156 2024-04-25T18:30:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 124/156 2024-04-25T18:30:10.129 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 125/156 2024-04-25T18:30:10.166 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T18:30:11.147 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T18:30:11.148 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T18:30:11.149 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T18:30:11.150 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T18:30:11.151 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T18:30:11.152 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T18:30:11.153 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T18:30:11.154 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T18:30:11.155 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T18:30:11.156 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T18:30:11.157 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T18:30:11.158 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T18:30:11.159 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:11.160 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T18:30:11.408 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T18:30:11.409 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T18:30:11.410 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T18:30:11.411 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T18:30:11.412 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T18:30:11.413 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T18:30:11.414 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T18:30:11.415 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T18:30:11.416 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T18:30:11.417 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T18:30:11.418 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T18:30:11.419 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T18:30:11.419 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T18:30:11.419 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:11.419 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T18:30:11.524 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 126/156 2024-04-25T18:30:11.576 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:30:12.742 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 127/156 2024-04-25T18:30:12.776 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:30:12.927 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 128/156 2024-04-25T18:30:13.117 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 129/156 2024-04-25T18:30:13.153 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 130/156 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T18:30:13.236 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T18:30:13.249 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.250 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.251 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T18:30:13.252 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.253 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.254 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T18:30:13.255 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T18:30:13.256 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T18:30:13.257 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T18:30:13.258 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T18:30:13.259 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:30:13.260 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-04-25T18:30:13.295 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:30:13.335 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 131/156 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T18:30:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T18:30:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T18:30:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T18:30:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T18:30:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T18:30:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-25T18:30:13.479 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-25T18:30:13.481 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 132/156 2024-04-25T18:30:13.564 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 133/156 2024-04-25T18:30:13.660 DEBUG:teuthology.parallel:result is None 2024-04-25T18:30:13.842 DEBUG:teuthology.parallel:result is None 2024-04-25T18:30:14.676 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T18:30:14.715 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 135/156 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T18:30:14.778 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:14.841 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:30:14.926 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 136/156 2024-04-25T18:30:14.927 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:14.927 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T18:30:14.927 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:14.999 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 137/156 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T18:30:15.079 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 138/156 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T18:30:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:20.872 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 139/156 2024-04-25T18:30:20.929 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 140/156 2024-04-25T18:30:21.012 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 141/156 2024-04-25T18:30:21.059 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T18:30:21.108 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T18:30:21.510 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 144/156 2024-04-25T18:30:21.567 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T18:30:21.596 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 146/156 2024-04-25T18:30:22.662 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:22.710 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:22.771 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T18:30:24.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 148/156 2024-04-25T18:30:24.310 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T18:30:24.393 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T18:30:24.584 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T18:30:24.639 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T18:30:24.639 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:24.782 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T18:30:24.830 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:24.830 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:24.977 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T18:30:25.142 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:25.443 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:25.444 INFO:teuthology.orchestra.run.smithi063.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T18:30:25.444 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 1/156 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2/156 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 3/156 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 4/156 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3156.gd69f1 5/156 2024-04-25T18:30:31.569 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 6/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 7/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 8/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 9/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 10/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 11/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 12/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 13/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 14/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_ 15/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 16/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 17/156 2024-04-25T18:30:31.570 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_6 18/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86 19/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 20/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el 21/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7. 22/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_6 23/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 24/156 2024-04-25T18:30:31.571 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 25/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 26/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 27/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 28/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 29/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8. 30/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noa 31/156 2024-04-25T18:30:31.584 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.n 32/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69 33/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el 34/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 35/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.e 36/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 37/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 38/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 39/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T18:30:31.585 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T18:30:31.586 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T18:30:31.587 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T18:30:31.588 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T18:30:31.589 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T18:30:31.590 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T18:30:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T18:30:31.592 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 149/156 2024-04-25T18:30:31.593 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T18:30:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 151/156 2024-04-25T18:30:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T18:30:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T18:30:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T18:30:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T18:30:34.147 INFO:teuthology.orchestra.run.smithi063.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.148 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.149 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top-2:19.0.0-3156.gd69f13d8.el7.noarch 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T18:30:34.150 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T18:30:34.151 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T18:30:34.152 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T18:30:34.153 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T18:30:34.154 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T18:30:34.155 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T18:30:34.156 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T18:30:34.157 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-3156.gd69f13d8.el7.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T18:30:34.158 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T18:30:34.159 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:30:34.159 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-25T18:30:34.897 DEBUG:teuthology.parallel:result is None 2024-04-25T18:30:34.897 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:35.171 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T18:30:35.230 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.230 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.230 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T18:30:35.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:35.526 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T18:30:35.552 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.552 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.552 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T18:30:35.554 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:35.699 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T18:30:35.724 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.724 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3156.gd69f13d8.el7 2024-04-25T18:30:35.725 INFO:teuthology.task.install:The correct ceph version 19.0.0-3156.gd69f13d8 is installed. 2024-04-25T18:30:35.726 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T18:30:35.726 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:30:35.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T18:30:35.806 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:30:35.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T18:30:35.836 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:30:35.836 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T18:30:35.867 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T18:30:35.867 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:30:35.867 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T18:30:35.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T18:30:36.038 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:30:36.038 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T18:30:36.066 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T18:30:36.134 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:30:36.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T18:30:36.162 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T18:30:36.228 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T18:30:36.229 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:30:36.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T18:30:36.290 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T18:30:36.392 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:30:36.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T18:30:36.416 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T18:30:36.479 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:30:36.479 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T18:30:36.503 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T18:30:36.566 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T18:30:36.566 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:30:36.566 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T18:30:36.629 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T18:30:36.730 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:30:36.730 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T18:30:36.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T18:30:36.816 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:30:36.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T18:30:36.841 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T18:30:36.905 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T18:30:36.996 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T18:30:36.996 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:36.997 INFO:tasks.cephadm:Cluster fsid is e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:30:36.997 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T18:30:36.997 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.63', 'mon.b': '172.21.15.70', 'mon.c': '172.21.15.114'} 2024-04-25T18:30:36.997 INFO:tasks.cephadm:First mon is mon.a on smithi063 2024-04-25T18:30:36.997 INFO:tasks.cephadm:First mgr is x 2024-04-25T18:30:36.997 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T18:30:36.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-04-25T18:30:37.062 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-04-25T18:30:37.085 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2024-04-25T18:30:37.109 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T18:30:37.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:37.262 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'd69f13d827952bb07bb9b7973bca14f97d764d6f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78795/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3156-gd69f13d8', 'node_name': '172.21.5.39+adami09', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3156.gd69f13d8'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-22 12:00:06.590653', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiescer', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-25T18:30:37.487 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-lusov-quiescer, sha1 d69f13d827952bb07bb9b7973bca14f97d764d6f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:30:37.488 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm 2024-04-25T18:30:37.488 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:38.206 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 18:30 /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:38.207 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:38.826 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 18:30 /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:38.827 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-lusov-quiescer/d69f13d827952bb07bb9b7973bca14f97d764d6f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:39.250 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-r--r--. 1 ubuntu ubuntu 781541 Apr 25 18:30 /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:39.250 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:39.297 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:39.316 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T18:30:39.349 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f on all hosts... 2024-04-25T18:30:39.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T18:30:39.354 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T18:30:39.359 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f pull 2024-04-25T18:30:40.946 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T18:30:40.956 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T18:30:41.600 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T18:31:12.359 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2024-04-25T18:31:12.360 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T18:31:13.089 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-25T18:31:13.090 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev)", 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76", 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4" 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T18:32:20.203 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:32:20.270 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-04-25T18:32:20.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-04-25T18:32:20.399 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2024-04-25T18:32:20.430 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-04-25T18:32:20.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-04-25T18:32:20.549 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2024-04-25T18:32:20.576 INFO:tasks.cephadm:Writing seed config... 2024-04-25T18:32:20.577 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T18:32:20.578 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-25T18:32:20.579 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T18:32:20.580 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T18:32:20.581 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:32:20.581 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T18:32:20.641 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e90c993c-0331-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T18:32:20.641 DEBUG:teuthology.orchestra.run.smithi063:mon.a> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T18:32:20.688 DEBUG:teuthology.orchestra.run.smithi063:mgr.x> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T18:32:20.735 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T18:32:20.735 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f -v bootstrap --fsid e90c993c-0331-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.63 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T18:32:21.007 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-25T18:32:21.007 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f', '-v', 'bootstrap', '--fsid', 'e90c993c-0331-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.63', '--skip-admin-label'] 2024-04-25T18:32:21.008 INFO:teuthology.orchestra.run.smithi063.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T18:32:21.008 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying podman|docker is present... 2024-04-25T18:32:21.069 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T18:32:21.069 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying lvm2 is present... 2024-04-25T18:32:21.069 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying time synchronization is in place... 2024-04-25T18:32:21.086 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T18:32:21.086 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T18:32:21.100 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T18:32:21.101 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:32:21.117 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-04-25T18:32:21.136 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-04-25T18:32:21.137 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-04-25T18:32:21.137 INFO:teuthology.orchestra.run.smithi063.stdout:Repeating the final host check... 2024-04-25T18:32:21.190 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T18:32:21.190 INFO:teuthology.orchestra.run.smithi063.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T18:32:21.190 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl is present 2024-04-25T18:32:21.190 INFO:teuthology.orchestra.run.smithi063.stdout:lvcreate is present 2024-04-25T18:32:21.206 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T18:32:21.206 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T18:32:21.225 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T18:32:21.225 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:32:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-04-25T18:32:21.266 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-04-25T18:32:21.267 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-04-25T18:32:21.267 INFO:teuthology.orchestra.run.smithi063.stdout:Host looks OK 2024-04-25T18:32:21.267 INFO:teuthology.orchestra.run.smithi063.stdout:Cluster fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:21.268 INFO:teuthology.orchestra.run.smithi063.stdout:Acquiring lock 140285087272912 on /run/cephadm/e90c993c-0331-11ef-bc93-c7b262605968.lock 2024-04-25T18:32:21.268 INFO:teuthology.orchestra.run.smithi063.stdout:Lock 140285087272912 acquired on /run/cephadm/e90c993c-0331-11ef-bc93-c7b262605968.lock 2024-04-25T18:32:21.269 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 3300 ... 2024-04-25T18:32:21.270 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 6789 ... 2024-04-25T18:32:21.270 INFO:teuthology.orchestra.run.smithi063.stdout:Base mon IP(s) is [172.21.15.63:3300, 172.21.15.63:6789], mon addrv is [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-04-25T18:32:21.283 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.63 metric 100 2024-04-25T18:32:21.283 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.63 metric 100 2024-04-25T18:32:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T18:32:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-25T18:32:21.294 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link 2024-04-25T18:32:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T18:32:21.309 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-04-25T18:32:21.309 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-04-25T18:32:21.309 INFO:teuthology.orchestra.run.smithi063.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T18:32:21.312 INFO:teuthology.orchestra.run.smithi063.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T18:32:21.314 INFO:teuthology.orchestra.run.smithi063.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T18:32:22.214 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f... 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:7f03586bcf9c6a2b4d690c118c4aa5b771f9c6f252d75a20fb283dd07093a738 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:426fede9fffa2c7dfe40a4f9322a34794b72405170134d54d2c30d9cc6c6a5be 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying config sha256:da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 2024-04-25T18:32:22.215 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T18:32:23.757 INFO:teuthology.orchestra.run.smithi063.stdout:ceph: stdout ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T18:32:23.757 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph version: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev) 2024-04-25T18:32:23.757 INFO:teuthology.orchestra.run.smithi063.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T18:32:25.174 INFO:teuthology.orchestra.run.smithi063.stdout:stat: stdout 167 167 2024-04-25T18:32:25.174 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial keys... 2024-04-25T18:32:26.279 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQC5oSpmKlnCJRAALqq7QJ1lS97a5l1GPBJsew== 2024-04-25T18:32:27.524 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQC6oSpmpveYLxAA3nnbemt0M+zpgR9+TMxLPg== 2024-04-25T18:32:35.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQC/oSpmyYoWDhAAv0iBc7ABA33Kw1Fv/Dz5qg== 2024-04-25T18:32:35.922 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial monmap... 2024-04-25T18:32:37.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool for a [v2:172.21.15.63:3300,v1:172.21.15.63:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: set fsid to e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:32:37.118 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mon... 2024-04-25T18:32:38.298 INFO:teuthology.orchestra.run.smithi063.stdout:create mon.a on 2024-04-25T18:32:38.704 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T18:32:38.953 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T18:32:39.265 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target. 2024-04-25T18:32:39.266 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target. 2024-04-25T18:32:39.657 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a 2024-04-25T18:32:39.657 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T18:32:39.943 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target.wants/ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968@.service. 2024-04-25T18:32:40.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:40 smithi063 podman[27623]: 2024-04-25 18:32:40.285070142 +0000 UTC m=+0.042049160 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:32:41.141 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:40 smithi063 podman[27623]: 2024-04-25 18:32:40.889961851 +0000 UTC m=+0.646940897 container create 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T18:32:41.141 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 podman[27623]: 2024-04-25 18:32:41.129471329 +0000 UTC m=+0.886450429 container init 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-25T18:32:41.142 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 podman[27623]: 2024-04-25 18:32:41.140792134 +0000 UTC m=+0.897771187 container start 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T18:32:41.287 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T18:32:41.288 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon to start... 2024-04-25T18:32:41.303 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon... 2024-04-25T18:32:41.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 ceph-mon[27642]: mkfs e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:41.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 ceph-mon[27642]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T18:32:41.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 bash[27623]: 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 2024-04-25T18:32:41.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:41 smithi063 systemd[1]: Started Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: monmap epoch 1 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: last_changed 2024-04-25T18:32:36.435483+0000 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: min_mon_release 19 (squid) 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: election_strategy: 1 2024-04-25T18:32:42.349 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:32:42.350 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: fsmap 2024-04-25T18:32:42.350 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T18:32:42.350 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: mgrmap e1: no daemons active 2024-04-25T18:32:42.350 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:42 smithi063 ceph-mon[27642]: from='client.? 172.21.15.63:0/665633364' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T18:32:42.656 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout id: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout services: 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.816306s) 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T18:32:42.658 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout data: 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T18:32:42.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.941+0000 7fd222200700 1 Processor -- start 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.941+0000 7fd222200700 1 -- start start 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.942+0000 7fd222200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.942+0000 7fd222200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c107e30 con 0x7fd21c107440 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.943+0000 7fd21b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.943+0000 7fd21b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46244/0 (socket says 172.21.15.63:46244) 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.943+0000 7fd21b400700 1 -- 172.21.15.63:0/4258039368 learned_addr learned my addr 172.21.15.63:0/4258039368 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.944+0000 7fd21b400700 1 -- 172.21.15.63:0/4258039368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21c108640 con 0x7fd21c107440 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.944+0000 7fd21b400700 1 --2- 172.21.15.63:0/4258039368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0098a0 tx=0x7fd20c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d846734694e4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.944+0000 7fd21a000700 1 -- 172.21.15.63:0/4258039368 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd20c004160 con 0x7fd21c107440 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.944+0000 7fd21a000700 1 -- 172.21.15.63:0/4258039368 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c004300 con 0x7fd21c107440 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.944+0000 7fd21a000700 1 -- 172.21.15.63:0/4258039368 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd20c0036a0 con 0x7fd21c107440 2024-04-25T18:32:42.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 -- 172.21.15.63:0/4258039368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 msgr2=0x7fd21c107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 --2- 172.21.15.63:0/4258039368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd20c0098a0 tx=0x7fd20c009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 -- 172.21.15.63:0/4258039368 shutdown_connections 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 --2- 172.21.15.63:0/4258039368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c107860 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 -- 172.21.15.63:0/4258039368 >> 172.21.15.63:0/4258039368 conn(0x7fd21c102a40 msgr2=0x7fd21c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 -- 172.21.15.63:0/4258039368 shutdown_connections 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.945+0000 7fd222200700 1 -- 172.21.15.63:0/4258039368 wait complete. 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.946+0000 7fd222200700 1 Processor -- start 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd222200700 1 -- start start 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd222200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd222200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c108b70 con 0x7fd21c107440 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd21b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:42.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd21b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46254/0 (socket says 172.21.15.63:46254) 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.947+0000 7fd21b400700 1 -- 172.21.15.63:0/665633364 learned_addr learned my addr 172.21.15.63:0/665633364 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd21b400700 1 -- 172.21.15.63:0/665633364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21c19c190 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd21b400700 1 --2- 172.21.15.63:0/665633364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd20c000c00 tx=0x7fd20c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd20c003db0 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c011630 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd20c011950 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.948+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21c19c420 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.949+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21c19f110 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.949+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd20c011af0 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.949+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd20c01c450 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.950+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21c0640a0 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:41.953+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd20c032020 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.034+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fd21c02e180 con 0x7fd21c107440 2024-04-25T18:32:42.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.035+0000 7fd213e00700 1 -- 172.21.15.63:0/665633364 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fd20c004260 con 0x7fd21c107440 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.036+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 msgr2=0x7fd21c19bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.036+0000 7fd222200700 1 --2- 172.21.15.63:0/665633364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd20c000c00 tx=0x7fd20c004750 comp rx=0 tx=0).stop 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.037+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 shutdown_connections 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.037+0000 7fd222200700 1 --2- 172.21.15.63:0/665633364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c107440 0x7fd21c19bc50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.037+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 >> 172.21.15.63:0/665633364 conn(0x7fd21c102a40 msgr2=0x7fd21c1928b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.037+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 shutdown_connections 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:42.037+0000 7fd222200700 1 -- 172.21.15.63:0/665633364 wait complete. 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:mon is available 2024-04-25T18:32:42.668 INFO:teuthology.orchestra.run.smithi063.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T18:32:43.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:43 smithi063 ceph-mon[27642]: from='client.? 172.21.15.63:0/2880705545' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T18:32:43.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:43 smithi063 ceph-mon[27642]: from='client.? 172.21.15.63:0/2880705545' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T18:32:44.161 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:44.162 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T18:32:44.162 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.360+0000 7efefb000700 1 Processor -- start 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.360+0000 7efefb000700 1 -- start start 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.361+0000 7efefb000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:44.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.361+0000 7efefb000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efef4110c10 con 0x7efef4110220 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.361+0000 7efef3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.361+0000 7efef3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46266/0 (socket says 172.21.15.63:46266) 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.361+0000 7efef3e00700 1 -- 172.21.15.63:0/393890539 learned_addr learned my addr 172.21.15.63:0/393890539 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.362+0000 7efef3e00700 1 -- 172.21.15.63:0/393890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef410ece0 con 0x7efef4110220 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.362+0000 7efef3e00700 1 --2- 172.21.15.63:0/393890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efee40098a0 tx=0x7efee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaccc067a5a750ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efef2a00700 1 -- 172.21.15.63:0/393890539 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee4004160 con 0x7efef4110220 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efef2a00700 1 -- 172.21.15.63:0/393890539 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4004300 con 0x7efef4110220 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efef2a00700 1 -- 172.21.15.63:0/393890539 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee40036a0 con 0x7efef4110220 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efefb000700 1 -- 172.21.15.63:0/393890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 msgr2=0x7efef4110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efefb000700 1 --2- 172.21.15.63:0/393890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efee40098a0 tx=0x7efee4009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efefb000700 1 -- 172.21.15.63:0/393890539 shutdown_connections 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efefb000700 1 --2- 172.21.15.63:0/393890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef4110640 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.363+0000 7efefb000700 1 -- 172.21.15.63:0/393890539 >> 172.21.15.63:0/393890539 conn(0x7efef407de20 msgr2=0x7efef4080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:44.169 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.364+0000 7efefb000700 1 -- 172.21.15.63:0/393890539 shutdown_connections 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.364+0000 7efefb000700 1 -- 172.21.15.63:0/393890539 wait complete. 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.365+0000 7efefb000700 1 Processor -- start 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.365+0000 7efefb000700 1 -- start start 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.365+0000 7efefb000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.366+0000 7efefb000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efef410f1b0 con 0x7efef4110220 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.366+0000 7efef3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.366+0000 7efef3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46268/0 (socket says 172.21.15.63:46268) 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.366+0000 7efef3e00700 1 -- 172.21.15.63:0/2880705545 learned_addr learned my addr 172.21.15.63:0/2880705545 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.366+0000 7efef3e00700 1 -- 172.21.15.63:0/2880705545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef41a28b0 con 0x7efef4110220 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efef3e00700 1 --2- 172.21.15.63:0/2880705545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee4003db0 con 0x7efef4110220 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4011630 con 0x7efef4110220 2024-04-25T18:32:44.170 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef41a2b40 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efee4011950 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.367+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef41a5830 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.368+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efee401a460 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.368+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efee401b780 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.369+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef40640a0 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.371+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efee4031020 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.450+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7efef402e170 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.458+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+492 (secure 0 0 0) 0x7efee401b9d0 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.459+0000 7efef0c00700 1 -- 172.21.15.63:0/2880705545 <== mon.0 v2:172.21.15.63:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efee401ad20 con 0x7efef4110220 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.460+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 msgr2=0x7efef41a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.460+0000 7efefb000700 1 --2- 172.21.15.63:0/2880705545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efee4000c00 tx=0x7efee4004750 comp rx=0 tx=0).stop 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.461+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 shutdown_connections 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.461+0000 7efefb000700 1 --2- 172.21.15.63:0/2880705545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efef4110220 0x7efef41a2370 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.461+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 >> 172.21.15.63:0/2880705545 conn(0x7efef407de20 msgr2=0x7efef4198f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:44.171 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.461+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 shutdown_connections 2024-04-25T18:32:44.172 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:43.461+0000 7efefb000700 1 -- 172.21.15.63:0/2880705545 wait complete. 2024-04-25T18:32:44.172 INFO:teuthology.orchestra.run.smithi063.stdout:Generating new minimal ceph.conf... 2024-04-25T18:32:45.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:44 smithi063 ceph-mon[27642]: from='client.? 172.21.15.63:0/1040978914' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.777+0000 7fca9e800700 1 Processor -- start 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.777+0000 7fca9e800700 1 -- start start 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.778+0000 7fca9e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.778+0000 7fca9e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca98110c10 con 0x7fca98110220 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.778+0000 7fca97400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.778+0000 7fca97400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46282/0 (socket says 172.21.15.63:46282) 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.778+0000 7fca97400700 1 -- 172.21.15.63:0/2677950037 learned_addr learned my addr 172.21.15.63:0/2677950037 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.779+0000 7fca97400700 1 -- 172.21.15.63:0/2677950037 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9810ece0 con 0x7fca98110220 2024-04-25T18:32:45.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.779+0000 7fca97400700 1 --2- 172.21.15.63:0/2677950037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca880098a0 tx=0x7fca88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65008449f5d9cb3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.779+0000 7fca96000700 1 -- 172.21.15.63:0/2677950037 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca88004160 con 0x7fca98110220 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.779+0000 7fca96000700 1 -- 172.21.15.63:0/2677950037 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fca88004300 con 0x7fca98110220 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.779+0000 7fca96000700 1 -- 172.21.15.63:0/2677950037 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca88011af0 con 0x7fca98110220 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 -- 172.21.15.63:0/2677950037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 msgr2=0x7fca98110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 --2- 172.21.15.63:0/2677950037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fca880098a0 tx=0x7fca88009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 -- 172.21.15.63:0/2677950037 shutdown_connections 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 --2- 172.21.15.63:0/2677950037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca98110640 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 -- 172.21.15.63:0/2677950037 >> 172.21.15.63:0/2677950037 conn(0x7fca9807de20 msgr2=0x7fca98080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 -- 172.21.15.63:0/2677950037 shutdown_connections 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.780+0000 7fca9e800700 1 -- 172.21.15.63:0/2677950037 wait complete. 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.781+0000 7fca9e800700 1 Processor -- start 2024-04-25T18:32:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca9e800700 1 -- start start 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca9e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca9e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca98113140 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca97400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca97400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46294/0 (socket says 172.21.15.63:46294) 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.782+0000 7fca97400700 1 -- 172.21.15.63:0/1040978914 learned_addr learned my addr 172.21.15.63:0/1040978914 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca97400700 1 -- 172.21.15.63:0/1040978914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca981a3160 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca97400700 1 --2- 172.21.15.63:0/1040978914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fca880117b0 tx=0x7fca88011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca88003af0 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fca88020070 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca981a33f0 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.783+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca88019630 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.784+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca981a60e0 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.784+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fca88019c30 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.784+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fca8800b3e0 con 0x7fca98110220 2024-04-25T18:32:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.785+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca980640a0 con 0x7fca98110220 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.788+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fca880042a0 con 0x7fca98110220 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.867+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fca981a6600 con 0x7fca98110220 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.867+0000 7fca8fe00700 1 -- 172.21.15.63:0/1040978914 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fca8801a770 con 0x7fca98110220 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.869+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 msgr2=0x7fca981a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.869+0000 7fca9e800700 1 --2- 172.21.15.63:0/1040978914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fca880117b0 tx=0x7fca88011b70 comp rx=0 tx=0).stop 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.869+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 shutdown_connections 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.869+0000 7fca9e800700 1 --2- 172.21.15.63:0/1040978914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca98110220 0x7fca981a2c20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.869+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 >> 172.21.15.63:0/1040978914 conn(0x7fca9807de20 msgr2=0x7fca9807f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.870+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 shutdown_connections 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:44.870+0000 7fca9e800700 1 -- 172.21.15.63:0/1040978914 wait complete. 2024-04-25T18:32:45.540 INFO:teuthology.orchestra.run.smithi063.stdout:Restarting the monitor... 2024-04-25T18:32:45.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:45 smithi063 systemd[1]: Stopping Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:32:46.314 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a[27638]: 2024-04-25T18:32:46.045+0000 7f7d02a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T18:32:46.314 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a[27638]: 2024-04-25T18:32:46.045+0000 7f7d02a00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T18:32:46.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:46 smithi063 podman[27883]: 2024-04-25 18:32:46.31325949 +0000 UTC m=+0.406002560 container died 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416) 2024-04-25T18:32:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:46 smithi063 podman[27883]: 2024-04-25 18:32:46.729094415 +0000 UTC m=+0.821837492 container cleanup 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T18:32:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:46 smithi063 bash[27883]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a 2024-04-25T18:32:47.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:47 smithi063 podman[27897]: 2024-04-25 18:32:47.129735575 +0000 UTC m=+0.803351924 container remove 471a5a6a0af1ebf3fb081fd8eaa57187dfec4304b33456a66a60c2b7d7e19825 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T18:32:48.098 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:47 smithi063 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T18:32:48.098 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:47 smithi063 systemd[1]: Stopped Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:32:48.098 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:47 smithi063 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.078s CPU time. 2024-04-25T18:32:48.098 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:47 smithi063 systemd[1]: Starting Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:32:48.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 podman[27986]: 2024-04-25 18:32:48.0969833 +0000 UTC m=+0.118865582 container create 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True) 2024-04-25T18:32:48.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 podman[27986]: 2024-04-25 18:32:48.011334349 +0000 UTC m=+0.033216608 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:32:48.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 podman[27986]: 2024-04-25 18:32:48.325733546 +0000 UTC m=+0.347615841 container init 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T18:32:48.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 podman[27986]: 2024-04-25 18:32:48.335785008 +0000 UTC m=+0.357667303 container start 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T18:32:48.427 INFO:teuthology.orchestra.run.smithi063.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T18:32:48.760 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: pidfile_write: ignore empty --pid-file 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: load: jerasure load: lrc 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: RocksDB version: 7.9.2 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Git sha 0 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: DB SUMMARY 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: DB Session ID: Q3D48LTBVR7CLJRY7T6W 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: CURRENT file: CURRENT 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T18:32:48.761 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105444 ; 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.error_if_exists: 0 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.create_if_missing: 0 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.paranoid_checks: 1 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.env: 0x55eb8361bcc0 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.info_log: 0x55eb9f07fc40 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.statistics: (nil) 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.use_fsync: 0 2024-04-25T18:32:48.762 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_log_file_size: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_fallocate: 1 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.use_direct_reads: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.db_log_dir: 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.wal_dir: 2024-04-25T18:32:48.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.write_buffer_manager: 0x55eb9f0130e0 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T18:32:48.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.unordered_write: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.row_cache: None 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.wal_filter: None 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T18:32:48.766 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.two_write_queues: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.wal_compression: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.atomic_flush: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.log_readahead_size: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T18:32:48.767 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_background_jobs: 2 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_background_compactions: -1 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_subcompactions: 1 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T18:32:48.768 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_open_files: -1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_background_flushes: -1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Compression algorithms supported: 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kZSTD supported: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kXpressCompression supported: 0 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kZlibCompression supported: 1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kSnappyCompression supported: 1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kLZ4Compression supported: 1 2024-04-25T18:32:48.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: kBZip2Compression supported: 0 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.merge_operator: 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_filter: None 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb9f07fe00) 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T18:32:48.770 INFO:journalctl@ceph.mon.a.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_type: 0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: data_block_index_type: 0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_shortening: 1 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: checksum: 4 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: no_block_cache: 0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache: 0x55eb9f0451f0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_options: 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: capacity : 536870912 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: num_shard_bits : 4 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: strict_capacity_limit : 0 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-04-25T18:32:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_cache_compressed: (nil) 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: persistent_cache: (nil) 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_size: 4096 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_size_deviation: 10 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_restart_interval: 16 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: index_block_restart_interval: 1 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: metadata_block_size: 4096 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: partition_filters: 0 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: use_delta_encoding: 1 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: filter_policy: bloomfilter 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: whole_key_filtering: 1 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: verify_compression: 0 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: format_version: 5 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: enable_index_compression: 1 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: block_align: 0 2024-04-25T18:32:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout: max_auto_readahead_size: 262144 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout: prepopulate_block_cache: 0 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout: initial_auto_readahead_size: 8192 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression: NoCompression 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.num_levels: 7 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T18:32:48.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T18:32:48.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T18:32:48.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T18:32:48.776 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.inplace_update_support: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.bloom_locality: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.max_successive_merges: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T18:32:48.777 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.ttl: 2592000 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enable_blob_files: false 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.min_blob_size: 0 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T18:32:48.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c433fd7e-e099-403b-af79-6130628a5c2e 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714069968391113, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714069968393341, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98441, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14264, "raw_average_key_size": 49, "raw_value_size": 90695, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714069968, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c433fd7e-e099-403b-af79-6130628a5c2e", "db_session_id": "Q3D48LTBVR7CLJRY7T6W", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714069968393492, "job": 1, "event": "recovery_finished"} 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb9f15a000 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: DB pointer 0x55eb9f144000 2024-04-25T18:32:48.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** DB Stats ** 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** Compaction Stats [default] ** 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: L0 2/0 100.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Sum 2/0 100.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-25T18:32:48.780 INFO:journalctl@ceph.mon.a.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** Compaction Stats [default] ** 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 63.4 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Cumulative compaction: 0.00 GB write, 13.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T18:32:48.781 INFO:journalctl@ceph.mon.a.smithi063.stdout: Interval compaction: 0.00 GB write, 13.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: Block cache BinnedLRUCache@0x55eb9f0451f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: starting mon.a rank 0 at public addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] at bind addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???) e1 preinit fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).mds e1 new map 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).mds e1 print_map 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: e1 2024-04-25T18:32:48.782 INFO:journalctl@ceph.mon.a.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout: legacy client fscid: -1 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout: 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout: No filesystems configured 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 bash[27986]: 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 systemd[1]: Started Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: monmap epoch 1 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: last_changed 2024-04-25T18:32:36.435483+0000 2024-04-25T18:32:48.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: min_mon_release 19 (squid) 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: election_strategy: 1 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: fsmap 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T18:32:48.784 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:48 smithi063 ceph-mon[28006]: mgrmap e1: no daemons active 2024-04-25T18:32:50.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.300+0000 7f0d16600700 1 Processor -- start 2024-04-25T18:32:50.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.300+0000 7f0d16600700 1 -- start start 2024-04-25T18:32:50.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.301+0000 7f0d16600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:50.134 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.301+0000 7f0d16600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d10063d40 con 0x7f0d10108fd0 2024-04-25T18:32:50.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.302+0000 7f0d0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:50.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.302+0000 7f0d0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34072/0 (socket says 172.21.15.63:34072) 2024-04-25T18:32:50.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.302+0000 7f0d0f400700 1 -- 172.21.15.63:0/280775869 learned_addr learned my addr 172.21.15.63:0/280775869 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:50.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.302+0000 7f0d0f400700 1 -- 172.21.15.63:0/280775869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1010b900 con 0x7f0d10108fd0 2024-04-25T18:32:50.135 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.303+0000 7f0d0f400700 1 --2- 172.21.15.63:0/280775869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98acf4d0a3814f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:50.139 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.303+0000 7f0d0e000700 1 -- 172.21.15.63:0/280775869 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d00004160 con 0x7f0d10108fd0 2024-04-25T18:32:50.139 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.303+0000 7f0d0e000700 1 -- 172.21.15.63:0/280775869 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d00004300 con 0x7f0d10108fd0 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.303+0000 7f0d0e000700 1 -- 172.21.15.63:0/280775869 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d00011af0 con 0x7f0d10108fd0 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 -- 172.21.15.63:0/280775869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 msgr2=0x7f0d1010b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 --2- 172.21.15.63:0/280775869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d00009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 -- 172.21.15.63:0/280775869 shutdown_connections 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 --2- 172.21.15.63:0/280775869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1010b3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 -- 172.21.15.63:0/280775869 >> 172.21.15.63:0/280775869 conn(0x7f0d10102a40 msgr2=0x7f0d10104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 -- 172.21.15.63:0/280775869 shutdown_connections 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.304+0000 7f0d16600700 1 -- 172.21.15.63:0/280775869 wait complete. 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.305+0000 7f0d16600700 1 Processor -- start 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d16600700 1 -- start start 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d16600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d16600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1010c850 con 0x7f0d10108fd0 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d0f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34078/0 (socket says 172.21.15.63:34078) 2024-04-25T18:32:50.140 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.306+0000 7f0d0f400700 1 -- 172.21.15.63:0/3015673111 learned_addr learned my addr 172.21.15.63:0/3015673111 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.307+0000 7f0d0f400700 1 -- 172.21.15.63:0/3015673111 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1019eb60 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.307+0000 7f0d0f400700 1 --2- 172.21.15.63:0/3015673111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d00011ac0 tx=0x7f0d00011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.307+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d00003b20 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.308+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1019ee50 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.307+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0d0001a8c0 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.308+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d000248d0 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.308+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d101a1ae0 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.308+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0d00022020 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.308+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0d0002dd90 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.309+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d100640a0 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.312+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0d0001aa70 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.391+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f0d101a2170 con 0x7f0d10108fd0 2024-04-25T18:32:50.141 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.394+0000 7f0d07e00700 1 -- 172.21.15.63:0/3015673111 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f0d0001a430 con 0x7f0d10108fd0 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.395+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 msgr2=0x7f0d1019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.395+0000 7f0d16600700 1 --2- 172.21.15.63:0/3015673111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d00011ac0 tx=0x7f0d00011ba0 comp rx=0 tx=0).stop 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.396+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 shutdown_connections 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.396+0000 7f0d16600700 1 --2- 172.21.15.63:0/3015673111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d10108fd0 0x7f0d1019e620 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.396+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 >> 172.21.15.63:0/3015673111 conn(0x7f0d10102a40 msgr2=0x7f0d101034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.396+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 shutdown_connections 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:49.396+0000 7f0d16600700 1 -- 172.21.15.63:0/3015673111 wait complete. 2024-04-25T18:32:50.142 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T18:32:50.164 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T18:32:50.164 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mgr... 2024-04-25T18:32:50.166 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T18:32:50.167 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T18:32:50.602 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x 2024-04-25T18:32:50.603 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T18:32:50.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:50 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3015673111' entity='client.admin' 2024-04-25T18:32:50.933 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968.target.wants/ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-e90c993c-0331-11ef-bc93-c7b262605968@.service. 2024-04-25T18:32:51.333 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:50 smithi063 systemd[1]: Starting Ceph mgr.x for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:32:51.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 podman[28232]: 2024-04-25 18:32:51.237276119 +0000 UTC m=+0.032825364 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f 2024-04-25T18:32:51.629 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 podman[28232]: 2024-04-25 18:32:51.35816975 +0000 UTC m=+0.153719055 container create fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T18:32:51.876 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T18:32:51.876 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-04-25T18:32:51.895 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T18:32:51.895 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:32:51.895 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-04-25T18:32:51.895 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T18:32:51.918 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T18:32:51.919 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-04-25T18:32:51.940 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T18:32:51.940 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:32:51.940 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-04-25T18:32:51.940 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T18:32:51.940 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr to start... 2024-04-25T18:32:51.941 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr... 2024-04-25T18:32:52.005 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 podman[28232]: 2024-04-25 18:32:51.74490054 +0000 UTC m=+0.540449826 container init fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T18:32:52.005 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 podman[28232]: 2024-04-25 18:32:51.75720205 +0000 UTC m=+0.552751302 container start fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T18:32:52.005 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 bash[28232]: fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 2024-04-25T18:32:52.005 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:51 smithi063 systemd[1]: Started Ceph mgr.x for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:32:52.277 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:52.003+0000 7fef7791d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:32:52.547 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:52.275+0000 7fef7791d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:32:52.547 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:52.414+0000 7fef7791d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:32:52.850 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/355125207' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T18:32:53.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:52.849+0000 7fef7791d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:32:53.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:52.966+0000 7fef7791d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:32:53.494 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:53.496 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-25T18:32:53.497 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T18:32:53.498 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T18:32:53.506 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T18:32:53.507 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T18:32:41.221555+0000", 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T18:32:53.508 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.604+0000 7f9c5c400700 1 Processor -- start 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.604+0000 7f9c5c400700 1 -- start start 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.605+0000 7f9c5c400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.605+0000 7f9c5c400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5407b7f0 con 0x7f9c5407cdc0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.605+0000 7f9c59a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.605+0000 7f9c59a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34114/0 (socket says 172.21.15.63:34114) 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.605+0000 7f9c59a00700 1 -- 172.21.15.63:0/1753132756 learned_addr learned my addr 172.21.15.63:0/1753132756 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.606+0000 7f9c59a00700 1 -- 172.21.15.63:0/1753132756 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5407b970 con 0x7f9c5407cdc0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.606+0000 7f9c59a00700 1 --2- 172.21.15.63:0/1753132756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098a0 tx=0x7f9c48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcbb4a23a5379d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.606+0000 7f9c53e00700 1 -- 172.21.15.63:0/1753132756 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c48004160 con 0x7f9c5407cdc0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.606+0000 7f9c53e00700 1 -- 172.21.15.63:0/1753132756 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9c48004300 con 0x7f9c5407cdc0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.606+0000 7f9c53e00700 1 -- 172.21.15.63:0/1753132756 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c48011af0 con 0x7f9c5407cdc0 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.607+0000 7f9c5c400700 1 -- 172.21.15.63:0/1753132756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 msgr2=0x7f9c5407b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:53.509 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.607+0000 7f9c5c400700 1 --2- 172.21.15.63:0/1753132756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9c480098a0 tx=0x7f9c48009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.607+0000 7f9c5c400700 1 -- 172.21.15.63:0/1753132756 shutdown_connections 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.607+0000 7f9c5c400700 1 --2- 172.21.15.63:0/1753132756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5407b220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.607+0000 7f9c5c400700 1 -- 172.21.15.63:0/1753132756 >> 172.21.15.63:0/1753132756 conn(0x7f9c54103b50 msgr2=0x7f9c54105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.608+0000 7f9c5c400700 1 -- 172.21.15.63:0/1753132756 shutdown_connections 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.608+0000 7f9c5c400700 1 -- 172.21.15.63:0/1753132756 wait complete. 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c5c400700 1 Processor -- start 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c5c400700 1 -- start start 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c5c400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c5c400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5410eac0 con 0x7f9c5407cdc0 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c59a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c59a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34124/0 (socket says 172.21.15.63:34124) 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.609+0000 7f9c59a00700 1 -- 172.21.15.63:0/355125207 learned_addr learned my addr 172.21.15.63:0/355125207 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c59a00700 1 -- 172.21.15.63:0/355125207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5419eb80 con 0x7f9c5407cdc0 2024-04-25T18:32:53.510 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c59a00700 1 --2- 172.21.15.63:0/355125207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c48003b20 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9c4801a8c0 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c48024940 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.610+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c5419ee10 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.611+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c541a1b00 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.611+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9c48022020 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.611+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9c4802dd90 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.612+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c540640a0 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.614+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9c48042e50 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.694+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9c541a2050 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.695+0000 7f9c52000700 1 -- 172.21.15.63:0/355125207 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9c48031020 con 0x7f9c5407cdc0 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.696+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 msgr2=0x7f9c5419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.696+0000 7f9c5c400700 1 --2- 172.21.15.63:0/355125207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9c48000c00 tx=0x7f9c48011b70 comp rx=0 tx=0).stop 2024-04-25T18:32:53.511 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.696+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 shutdown_connections 2024-04-25T18:32:53.512 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.696+0000 7f9c5c400700 1 --2- 172.21.15.63:0/355125207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c5407cdc0 0x7f9c5419e640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:53.512 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.696+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 >> 172.21.15.63:0/355125207 conn(0x7f9c54103b50 msgr2=0x7f9c54104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:53.512 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.697+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 shutdown_connections 2024-04-25T18:32:53.512 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:52.697+0000 7f9c5c400700 1 -- 172.21.15.63:0/355125207 wait complete. 2024-04-25T18:32:53.512 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (1/15)... 2024-04-25T18:32:54.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:54.610+0000 7fef7791d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:32:54.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:54.843+0000 7fef7791d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:32:55.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:55.063+0000 7fef7791d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:32:55.821 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:55.456+0000 7fef7791d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:32:55.821 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:55.589+0000 7fef7791d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:32:56.114 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:55.851+0000 7fef7791d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:32:56.114 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:56.112+0000 7fef7791d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:32:56.735 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:56 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2445831573' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T18:32:57.090 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:56.939+0000 7fef7791d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T18:32:57.107 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T18:32:57.108 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.109 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T18:32:57.109 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:57.109 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T18:32:57.109 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T18:32:57.109 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T18:32:57.117 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T18:32:57.118 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T18:32:41.221555+0000", 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.292+0000 7f0266400700 1 Processor -- start 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.292+0000 7f0266400700 1 -- start start 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.292+0000 7f0266400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.292+0000 7f0266400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0260063d20 con 0x7f0260106de0 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.293+0000 7f025f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:57.119 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.293+0000 7f025f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34132/0 (socket says 172.21.15.63:34132) 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.293+0000 7f025f400700 1 -- 172.21.15.63:0/3926161683 learned_addr learned my addr 172.21.15.63:0/3926161683 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.293+0000 7f025f400700 1 -- 172.21.15.63:0/3926161683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0260107740 con 0x7f0260106de0 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.294+0000 7f025f400700 1 --2- 172.21.15.63:0/3926161683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02500098a0 tx=0x7f0250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf591f0f2ad424b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.294+0000 7f025e000700 1 -- 172.21.15.63:0/3926161683 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0250004160 con 0x7f0260106de0 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.294+0000 7f025e000700 1 -- 172.21.15.63:0/3926161683 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0250004300 con 0x7f0260106de0 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.294+0000 7f025e000700 1 -- 172.21.15.63:0/3926161683 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0250011af0 con 0x7f0260106de0 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 -- 172.21.15.63:0/3926161683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 msgr2=0x7f0260107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 --2- 172.21.15.63:0/3926161683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02500098a0 tx=0x7f0250009bb0 comp rx=0 tx=0).stop 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 -- 172.21.15.63:0/3926161683 shutdown_connections 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 --2- 172.21.15.63:0/3926161683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f0260107200 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 -- 172.21.15.63:0/3926161683 >> 172.21.15.63:0/3926161683 conn(0x7f0260102a40 msgr2=0x7f0260104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 -- 172.21.15.63:0/3926161683 shutdown_connections 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.295+0000 7f0266400700 1 -- 172.21.15.63:0/3926161683 wait complete. 2024-04-25T18:32:57.120 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.296+0000 7f0266400700 1 Processor -- start 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.296+0000 7f0266400700 1 -- start start 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f0266400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f0266400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0260108770 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f025f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f025f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34134/0 (socket says 172.21.15.63:34134) 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f025f400700 1 -- 172.21.15.63:0/2445831573 learned_addr learned my addr 172.21.15.63:0/2445831573 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.297+0000 7f025f400700 1 -- 172.21.15.63:0/2445831573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026019eb50 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f025f400700 1 --2- 172.21.15.63:0/2445831573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0250011ac0 tx=0x7f0250011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0250003b20 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f025001a8c0 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026019ede0 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02500248d0 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02601a1ad0 con 0x7f0260106de0 2024-04-25T18:32:57.121 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0250022020 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.298+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f02500195d0 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.299+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02600640a0 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.304+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0250042e50 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.383+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f026002e180 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.384+0000 7f0257e00700 1 -- 172.21.15.63:0/2445831573 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0250031020 con 0x7f0260106de0 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 msgr2=0x7f026019e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 --2- 172.21.15.63:0/2445831573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0250011ac0 tx=0x7f0250011ba0 comp rx=0 tx=0).stop 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 shutdown_connections 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 --2- 172.21.15.63:0/2445831573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0260106de0 0x7f026019e610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 >> 172.21.15.63:0/2445831573 conn(0x7f0260102a40 msgr2=0x7f02601952f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.386+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 shutdown_connections 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:56.387+0000 7f0266400700 1 -- 172.21.15.63:0/2445831573 wait complete. 2024-04-25T18:32:57.122 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (2/15)... 2024-04-25T18:32:57.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:57.507+0000 7fef7791d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:32:57.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:57.636+0000 7fef7791d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:32:58.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:57.906+0000 7fef7791d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:32:58.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:58.585+0000 7fef7791d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:32:58.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:58.711+0000 7fef7791d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:32:59.363 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:59.039+0000 7fef7791d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:32:59.363 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:59.233+0000 7fef7791d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:32:59.363 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:59.361+0000 7fef7791d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:32:59.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:59.520+0000 7fef7791d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:32:59.931 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:32:59.899+0000 7fef7791d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:33:00.299 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:32:59 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3033490044' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T18:33:00.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T18:33:00.688 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:00.688 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T18:33:00.688 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T18:33:00.688 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T18:33:00.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:00.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T18:32:41.221555+0000", 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.836+0000 7fb0cdc00700 1 Processor -- start 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.836+0000 7fb0cdc00700 1 -- start start 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.837+0000 7fb0cdc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.837+0000 7fb0cdc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c807b7b0 con 0x7fb0c807cd80 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.837+0000 7fb0c6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:00.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.837+0000 7fb0c6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33094/0 (socket says 172.21.15.63:33094) 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.837+0000 7fb0c6a00700 1 -- 172.21.15.63:0/2904146075 learned_addr learned my addr 172.21.15.63:0/2904146075 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.838+0000 7fb0c6a00700 1 -- 172.21.15.63:0/2904146075 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c807b930 con 0x7fb0c807cd80 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.838+0000 7fb0c6a00700 1 --2- 172.21.15.63:0/2904146075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80098a0 tx=0x7fb0b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4833b5e5550813da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.838+0000 7fb0c5600700 1 -- 172.21.15.63:0/2904146075 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0b8004160 con 0x7fb0c807cd80 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.838+0000 7fb0c5600700 1 -- 172.21.15.63:0/2904146075 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb0b8004300 con 0x7fb0c807cd80 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.838+0000 7fb0c5600700 1 -- 172.21.15.63:0/2904146075 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0b8011af0 con 0x7fb0c807cd80 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.839+0000 7fb0cdc00700 1 -- 172.21.15.63:0/2904146075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 msgr2=0x7fb0c807b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.839+0000 7fb0cdc00700 1 --2- 172.21.15.63:0/2904146075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80098a0 tx=0x7fb0b8009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.839+0000 7fb0cdc00700 1 -- 172.21.15.63:0/2904146075 shutdown_connections 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.840+0000 7fb0cdc00700 1 --2- 172.21.15.63:0/2904146075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c807b1e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.840+0000 7fb0cdc00700 1 -- 172.21.15.63:0/2904146075 >> 172.21.15.63:0/2904146075 conn(0x7fb0c8103b50 msgr2=0x7fb0c8105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.840+0000 7fb0cdc00700 1 -- 172.21.15.63:0/2904146075 shutdown_connections 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.840+0000 7fb0cdc00700 1 -- 172.21.15.63:0/2904146075 wait complete. 2024-04-25T18:33:00.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.841+0000 7fb0cdc00700 1 Processor -- start 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.841+0000 7fb0cdc00700 1 -- start start 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.841+0000 7fb0cdc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.841+0000 7fb0cdc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c810eb00 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.842+0000 7fb0c6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.842+0000 7fb0c6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33102/0 (socket says 172.21.15.63:33102) 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.842+0000 7fb0c6a00700 1 -- 172.21.15.63:0/3033490044 learned_addr learned my addr 172.21.15.63:0/3033490044 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.842+0000 7fb0c6a00700 1 -- 172.21.15.63:0/3033490044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c819eb70 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.842+0000 7fb0c6a00700 1 --2- 172.21.15.63:0/3033490044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8011ac0 tx=0x7fb0b8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0b8003b20 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb0b801a8c0 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb0b80248d0 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c819ee00 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c81a1af0 con 0x7fb0c807cd80 2024-04-25T18:33:00.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb0b8022020 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.843+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb0b802dde0 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.844+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c80640a0 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.847+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb0b8042e50 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.929+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb0c81a2030 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.929+0000 7fb0bf400700 1 -- 172.21.15.63:0/3033490044 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fb0b8031020 con 0x7fb0c807cd80 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 msgr2=0x7fb0c819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 --2- 172.21.15.63:0/3033490044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8011ac0 tx=0x7fb0b8011ba0 comp rx=0 tx=0).stop 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 shutdown_connections 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 --2- 172.21.15.63:0/3033490044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0c807cd80 0x7fb0c819e630 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 >> 172.21.15.63:0/3033490044 conn(0x7fb0c8103b50 msgr2=0x7fb0c8104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.931+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 shutdown_connections 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:32:59.932+0000 7fb0cdc00700 1 -- 172.21.15.63:0/3033490044 wait complete. 2024-04-25T18:33:00.701 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (3/15)... 2024-04-25T18:33:00.983 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:00.693+0000 7fef7791d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:33:01.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:00.982+0000 7fef7791d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:33:01.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: Activating manager daemon x 2024-04-25T18:33:01.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: mgrmap e2: x(active, starting, since 0.0122536s) 2024-04-25T18:33:01.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:33:01.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: Manager daemon x is now available 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' 2024-04-25T18:33:01.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:01 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' 2024-04-25T18:33:02.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:02 smithi063 ceph-mon[28006]: from='mgr.14100 172.21.15.63:0/3483073840' entity='mgr.x' 2024-04-25T18:33:02.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:02 smithi063 ceph-mon[28006]: mgrmap e3: x(active, since 1.02289s) 2024-04-25T18:33:04.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:04 smithi063 ceph-mon[28006]: mgrmap e4: x(active, since 2s) 2024-04-25T18:33:04.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:04 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2597289179' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T18:33:04.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:33:04.699 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:04.700 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T18:33:04.702 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:04.703 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T18:33:04.703 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T18:33:04.703 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T18:33:04.704 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T18:33:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T18:33:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T18:33:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T18:33:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T18:33:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T18:33:04.707 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T18:33:04.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T18:33:04.708 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-04-25T18:33:04.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:33:04.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T18:33:04.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T18:33:04.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T18:32:41.221555+0000", 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.516+0000 7f34e9800700 1 Processor -- start 2024-04-25T18:33:04.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.516+0000 7f34e9800700 1 -- start start 2024-04-25T18:33:04.721 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.517+0000 7f34e9800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:04.721 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.517+0000 7f34e9800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f34e410f960 con 0x7f34e410ef70 2024-04-25T18:33:04.721 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.517+0000 7f34e2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:04.721 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.517+0000 7f34e2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33178/0 (socket says 172.21.15.63:33178) 2024-04-25T18:33:04.721 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.517+0000 7f34e2a00700 1 -- 172.21.15.63:0/1288679622 learned_addr learned my addr 172.21.15.63:0/1288679622 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.518+0000 7f34e2a00700 1 -- 172.21.15.63:0/1288679622 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e410fd30 con 0x7f34e410ef70 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.518+0000 7f34e2a00700 1 --2- 172.21.15.63:0/1288679622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34d40098a0 tx=0x7f34d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c9f6ae6d9d481b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.518+0000 7f34e1600700 1 -- 172.21.15.63:0/1288679622 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d4004160 con 0x7f34e410ef70 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.518+0000 7f34e1600700 1 -- 172.21.15.63:0/1288679622 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f34d4004300 con 0x7f34e410ef70 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.518+0000 7f34e1600700 1 -- 172.21.15.63:0/1288679622 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d4011af0 con 0x7f34e410ef70 2024-04-25T18:33:04.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 -- 172.21.15.63:0/1288679622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 msgr2=0x7f34e410f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 --2- 172.21.15.63:0/1288679622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34d40098a0 tx=0x7f34d4009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 -- 172.21.15.63:0/1288679622 shutdown_connections 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 --2- 172.21.15.63:0/1288679622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e410f390 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 -- 172.21.15.63:0/1288679622 >> 172.21.15.63:0/1288679622 conn(0x7f34e407de20 msgr2=0x7f34e4080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 -- 172.21.15.63:0/1288679622 shutdown_connections 2024-04-25T18:33:04.723 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.519+0000 7f34e9800700 1 -- 172.21.15.63:0/1288679622 wait complete. 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.520+0000 7f34e9800700 1 Processor -- start 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.520+0000 7f34e9800700 1 -- start start 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.521+0000 7f34e9800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.521+0000 7f34e9800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f34e41130e0 con 0x7f34e410ef70 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.521+0000 7f34e2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:04.724 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.521+0000 7f34e2a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33186/0 (socket says 172.21.15.63:33186) 2024-04-25T18:33:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.521+0000 7f34e2a00700 1 -- 172.21.15.63:0/2597289179 learned_addr learned my addr 172.21.15.63:0/2597289179 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34e2a00700 1 -- 172.21.15.63:0/2597289179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e41a2f90 con 0x7f34e410ef70 2024-04-25T18:33:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34e2a00700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f34d4003f40 tx=0x7f34d4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d40038b0 con 0x7f34e410ef70 2024-04-25T18:33:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f34d4003a50 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e41a3220 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34d401a5c0 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.522+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e41a5f10 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.523+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f34d4024430 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.524+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e40640a0 con 0x7f34e410ef70 2024-04-25T18:33:04.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.524+0000 7f34db400700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 0x7f34c803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.524+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f34d404da20 con 0x7f34e410ef70 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.524+0000 7f34e2000700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 0x7f34c803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.525+0000 7f34e2000700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 0x7f34c803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34d0006fd0 tx=0x7f34d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.531+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34d404d440 con 0x7f34e410ef70 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.853+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f34e41a60e0 con 0x7f34e410ef70 2024-04-25T18:33:04.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.854+0000 7f34db400700 1 -- 172.21.15.63:0/2597289179 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f34e41a60e0 con 0x7f34e410ef70 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.857+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 msgr2=0x7f34c803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.857+0000 7f34e9800700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 0x7f34c803ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34d0006fd0 tx=0x7f34d0006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.857+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 msgr2=0x7f34e41a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.857+0000 7f34e9800700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f34d4003f40 tx=0x7f34d4011890 comp rx=0 tx=0).stop 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 shutdown_connections 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f34c8039560 0x7f34c803ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 --2- 172.21.15.63:0/2597289179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34e410ef70 0x7f34e41a2a50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 >> 172.21.15.63:0/2597289179 conn(0x7f34e407de20 msgr2=0x7f34e407e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 shutdown_connections 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:03.858+0000 7f34e9800700 1 -- 172.21.15.63:0/2597289179 wait complete. 2024-04-25T18:33:04.728 INFO:teuthology.orchestra.run.smithi063.stdout:mgr is available 2024-04-25T18:33:06.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:05 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3558856400' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T18:33:06.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:33:06.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout bdev_async_discard = True 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T18:33:06.674 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.473+0000 7fbc21600700 1 Processor -- start 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.474+0000 7fbc21600700 1 -- start start 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.474+0000 7fbc21600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.474+0000 7fbc21600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c063ca0 con 0x7fbc1c109070 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.474+0000 7fbc1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.475+0000 7fbc1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33196/0 (socket says 172.21.15.63:33196) 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.475+0000 7fbc1aa00700 1 -- 172.21.15.63:0/1508720445 learned_addr learned my addr 172.21.15.63:0/1508720445 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.475+0000 7fbc1aa00700 1 -- 172.21.15.63:0/1508720445 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c10b9a0 con 0x7fbc1c109070 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.475+0000 7fbc1aa00700 1 --2- 172.21.15.63:0/1508720445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c0098a0 tx=0x7fbc0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eddf5e100fda48a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.476+0000 7fbc1a000700 1 -- 172.21.15.63:0/1508720445 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc0c004160 con 0x7fbc1c109070 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.476+0000 7fbc1a000700 1 -- 172.21.15.63:0/1508720445 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbc0c004300 con 0x7fbc1c109070 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.476+0000 7fbc1a000700 1 -- 172.21.15.63:0/1508720445 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc0c011af0 con 0x7fbc1c109070 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.476+0000 7fbc21600700 1 -- 172.21.15.63:0/1508720445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 msgr2=0x7fbc1c10b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.476+0000 7fbc21600700 1 --2- 172.21.15.63:0/1508720445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c0098a0 tx=0x7fbc0c009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:06.675 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.477+0000 7fbc21600700 1 -- 172.21.15.63:0/1508720445 shutdown_connections 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.477+0000 7fbc21600700 1 --2- 172.21.15.63:0/1508720445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c10b460 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.477+0000 7fbc21600700 1 -- 172.21.15.63:0/1508720445 >> 172.21.15.63:0/1508720445 conn(0x7fbc1c102a40 msgr2=0x7fbc1c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.477+0000 7fbc21600700 1 -- 172.21.15.63:0/1508720445 shutdown_connections 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.477+0000 7fbc21600700 1 -- 172.21.15.63:0/1508720445 wait complete. 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.478+0000 7fbc21600700 1 Processor -- start 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.478+0000 7fbc21600700 1 -- start start 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.478+0000 7fbc21600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.478+0000 7fbc21600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c10c8b0 con 0x7fbc1c109070 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.479+0000 7fbc1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.479+0000 7fbc1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33208/0 (socket says 172.21.15.63:33208) 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.479+0000 7fbc1aa00700 1 -- 172.21.15.63:0/3558856400 learned_addr learned my addr 172.21.15.63:0/3558856400 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.479+0000 7fbc1aa00700 1 -- 172.21.15.63:0/3558856400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c19ebd0 con 0x7fbc1c109070 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.479+0000 7fbc1aa00700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c011ac0 tx=0x7fbc0c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:06.676 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.480+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc0c003b20 con 0x7fbc1c109070 2024-04-25T18:33:06.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.480+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c19ee60 con 0x7fbc1c109070 2024-04-25T18:33:06.677 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.480+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbc0c01a8c0 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.480+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc0c0248d0 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.480+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c1a1b50 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.481+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fbc0c022020 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.481+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c0640a0 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.482+0000 7fbc13e00700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 0x7fbc0003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.482+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc0c04d600 con 0x7fbc1c109070 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.482+0000 7fbc12000700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 0x7fbc0003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:06.679 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.483+0000 7fbc12000700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 0x7fbc0003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc04006fd0 tx=0x7fbc04006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.487+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc0c02cdf0 con 0x7fbc1c109070 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.717+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbc1c1a1d20 con 0x7fbc1c109070 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.719+0000 7fbc13e00700 1 -- 172.21.15.63:0/3558856400 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+436 (secure 0 0 0) 0x7fbc1c1a1d20 con 0x7fbc1c109070 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 msgr2=0x7fbc0003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 0x7fbc0003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbc04006fd0 tx=0x7fbc04006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 msgr2=0x7fbc1c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c011ac0 tx=0x7fbc0c011ba0 comp rx=0 tx=0).stop 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 shutdown_connections 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fbc000395b0 0x7fbc0003ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 --2- 172.21.15.63:0/3558856400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc1c109070 0x7fbc1c19e690 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 >> 172.21.15.63:0/3558856400 conn(0x7fbc1c102a40 msgr2=0x7fbc1c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 shutdown_connections 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:05.722+0000 7fbc21600700 1 -- 172.21.15.63:0/3558856400 wait complete. 2024-04-25T18:33:06.680 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling cephadm module... 2024-04-25T18:33:07.997 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:07 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/188843790' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T18:33:07.997 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setuser ceph since I am not root 2024-04-25T18:33:07.997 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setgroup ceph since I am not root 2024-04-25T18:33:07.997 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:07.995+0000 7fa1cb0e5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:33:08.521 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:08.247+0000 7fa1cb0e5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:33:08.521 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:08.382+0000 7fa1cb0e5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.322+0000 7fd25d400700 1 Processor -- start 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.323+0000 7fd25d400700 1 -- start start 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.323+0000 7fd25d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.323+0000 7fd25d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd258110c10 con 0x7fd258110220 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.324+0000 7fd256800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.324+0000 7fd256800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33938/0 (socket says 172.21.15.63:33938) 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.324+0000 7fd256800700 1 -- 172.21.15.63:0/1264581612 learned_addr learned my addr 172.21.15.63:0/1264581612 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.324+0000 7fd256800700 1 -- 172.21.15.63:0/1264581612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25810ece0 con 0x7fd258110220 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.325+0000 7fd256800700 1 --2- 172.21.15.63:0/1264581612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd2480098a0 tx=0x7fd248009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf1762f8d483039 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.325+0000 7fd255400700 1 -- 172.21.15.63:0/1264581612 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd248004160 con 0x7fd258110220 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.325+0000 7fd255400700 1 -- 172.21.15.63:0/1264581612 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd248004300 con 0x7fd258110220 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.325+0000 7fd255400700 1 -- 172.21.15.63:0/1264581612 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd248011af0 con 0x7fd258110220 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 -- 172.21.15.63:0/1264581612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 msgr2=0x7fd258110640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 --2- 172.21.15.63:0/1264581612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd2480098a0 tx=0x7fd248009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 -- 172.21.15.63:0/1264581612 shutdown_connections 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 --2- 172.21.15.63:0/1264581612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd258110640 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 -- 172.21.15.63:0/1264581612 >> 172.21.15.63:0/1264581612 conn(0x7fd25807de20 msgr2=0x7fd258080240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 -- 172.21.15.63:0/1264581612 shutdown_connections 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.326+0000 7fd25d400700 1 -- 172.21.15.63:0/1264581612 wait complete. 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.327+0000 7fd25d400700 1 Processor -- start 2024-04-25T18:33:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.327+0000 7fd25d400700 1 -- start start 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd25d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd25d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd258113140 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd256800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd256800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33944/0 (socket says 172.21.15.63:33944) 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd256800700 1 -- 172.21.15.63:0/188843790 learned_addr learned my addr 172.21.15.63:0/188843790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.328+0000 7fd256800700 1 -- 172.21.15.63:0/188843790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2581a3160 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd256800700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd2480117b0 tx=0x7fd248011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd248003af0 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd248020070 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd248019630 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2581a33f0 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.329+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2581a60e0 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.330+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fd24801a9c0 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.330+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2580640a0 con 0x7fd258110220 2024-04-25T18:33:08.536 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.331+0000 7fd24f400700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 0x7fd23c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.331+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd24804db10 con 0x7fd258110220 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.331+0000 7fd255e00700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 0x7fd23c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.332+0000 7fd255e00700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 0x7fd23c03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd244006fd0 tx=0x7fd244006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.337+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd24800e260 con 0x7fd258110220 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.630+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd2581a6600 con 0x7fd258110220 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.684+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd2480042a0 con 0x7fd258110220 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.688+0000 7fd24f400700 1 -- 172.21.15.63:0/188843790 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd24801a430 con 0x7fd258110220 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 msgr2=0x7fd23c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 0x7fd23c03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd244006fd0 tx=0x7fd244006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 msgr2=0x7fd2581a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd2480117b0 tx=0x7fd248011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 shutdown_connections 2024-04-25T18:33:08.537 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7fd23c039560 0x7fd23c03ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:08.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 --2- 172.21.15.63:0/188843790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258110220 0x7fd2581a2c20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:08.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 >> 172.21.15.63:0/188843790 conn(0x7fd25807de20 msgr2=0x7fd25807f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:08.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.689+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 shutdown_connections 2024-04-25T18:33:08.538 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:07.690+0000 7fd25d400700 1 -- 172.21.15.63:0/188843790 wait complete. 2024-04-25T18:33:08.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/188843790' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T18:33:08.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-mon[28006]: mgrmap e5: x(active, since 6s) 2024-04-25T18:33:09.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:08.803+0000 7fa1cb0e5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:33:09.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:08.921+0000 7fa1cb0e5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:33:09.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:09 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3351381814' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T18:33:10.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.166+0000 7f907ca00700 1 Processor -- start 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.167+0000 7f907ca00700 1 -- start start 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.167+0000 7f907ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.167+0000 7f907ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f907807b7b0 con 0x7f907807cd80 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.168+0000 7f9075e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:10.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.168+0000 7f9075e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33986/0 (socket says 172.21.15.63:33986) 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.168+0000 7f9075e00700 1 -- 172.21.15.63:0/3201776358 learned_addr learned my addr 172.21.15.63:0/3201776358 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.168+0000 7f9075e00700 1 -- 172.21.15.63:0/3201776358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907807b930 con 0x7f907807cd80 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.168+0000 7f9075e00700 1 --2- 172.21.15.63:0/3201776358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90680098a0 tx=0x7f9068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4991eddef1242cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.169+0000 7f9074a00700 1 -- 172.21.15.63:0/3201776358 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9068004160 con 0x7f907807cd80 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.169+0000 7f9074a00700 1 -- 172.21.15.63:0/3201776358 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9068004300 con 0x7f907807cd80 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.169+0000 7f9074a00700 1 -- 172.21.15.63:0/3201776358 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9068011af0 con 0x7f907807cd80 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.169+0000 7f907ca00700 1 -- 172.21.15.63:0/3201776358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 msgr2=0x7f907807b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.169+0000 7f907ca00700 1 --2- 172.21.15.63:0/3201776358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f90680098a0 tx=0x7f9068009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.170+0000 7f907ca00700 1 -- 172.21.15.63:0/3201776358 shutdown_connections 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.170+0000 7f907ca00700 1 --2- 172.21.15.63:0/3201776358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907807b1e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.170+0000 7f907ca00700 1 -- 172.21.15.63:0/3201776358 >> 172.21.15.63:0/3201776358 conn(0x7f9078103b50 msgr2=0x7f9078105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.170+0000 7f907ca00700 1 -- 172.21.15.63:0/3201776358 shutdown_connections 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.170+0000 7f907ca00700 1 -- 172.21.15.63:0/3201776358 wait complete. 2024-04-25T18:33:10.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.171+0000 7f907ca00700 1 Processor -- start 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.171+0000 7f907ca00700 1 -- start start 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f907ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f907ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f907810eb00 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f9075e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f9075e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33998/0 (socket says 172.21.15.63:33998) 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f9075e00700 1 -- 172.21.15.63:0/3351381814 learned_addr learned my addr 172.21.15.63:0/3351381814 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.172+0000 7f9075e00700 1 -- 172.21.15.63:0/3351381814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907819eb70 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f9075e00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9068011ac0 tx=0x7f9068011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9068003b20 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f906801a8c0 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90680248d0 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907819ee00 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.173+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90781a1af0 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.174+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90780640a0 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.174+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f9068022020 con 0x7f907807cd80 2024-04-25T18:33:10.337 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.175+0000 7f906ea00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 0x7f905c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.175+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f906804d4d0 con 0x7f907807cd80 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.175+0000 7f9075400700 1 -- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 msgr2=0x7f905c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.175+0000 7f9075400700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 0x7f905c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.181+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f906801ab20 con 0x7f907807cd80 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.376+0000 7f9075400700 1 -- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 msgr2=0x7f905c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.376+0000 7f9075400700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 0x7f905c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.484+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f90781a1cc0 con 0x7f907807cd80 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.486+0000 7f906ea00700 1 -- 172.21.15.63:0/3351381814 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f90781a1cc0 con 0x7f907807cd80 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 msgr2=0x7f905c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 0x7f905c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 msgr2=0x7f907819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9068011ac0 tx=0x7f9068011ba0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 shutdown_connections 2024-04-25T18:33:10.338 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f905c039600 0x7f905c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 --2- 172.21.15.63:0/3351381814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f907807cd80 0x7f907819e630 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.490+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 >> 172.21.15.63:0/3351381814 conn(0x7f9078103b50 msgr2=0x7f9078104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.494+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 shutdown_connections 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:09.494+0000 7f907ca00700 1 -- 172.21.15.63:0/3351381814 wait complete. 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-04-25T18:33:10.339 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 5... 2024-04-25T18:33:10.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:10 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:10.558+0000 7fa1cb0e5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:33:10.963 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:10 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:10.801+0000 7fa1cb0e5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:33:11.236 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:11 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:11.023+0000 7fa1cb0e5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:33:11.545 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:11 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:11.418+0000 7fa1cb0e5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:33:11.545 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:11 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:11.543+0000 7fa1cb0e5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:33:12.066 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:11 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:11.802+0000 7fa1cb0e5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:33:12.066 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:12 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:12.064+0000 7fa1cb0e5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:33:13.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:12 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:12.895+0000 7fa1cb0e5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:33:13.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:13 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:13.470+0000 7fa1cb0e5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:33:13.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:13 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:13.601+0000 7fa1cb0e5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:33:14.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:13 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:13.876+0000 7fa1cb0e5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:33:14.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:14.566+0000 7fa1cb0e5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:33:14.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:14.693+0000 7fa1cb0e5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:33:15.342 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:15.027+0000 7fa1cb0e5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:33:15.343 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:15.211+0000 7fa1cb0e5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:33:15.343 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:15.341+0000 7fa1cb0e5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:33:15.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:15.497+0000 7fa1cb0e5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:33:16.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:15.873+0000 7fa1cb0e5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:33:16.954 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:16.658+0000 7fa1cb0e5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:16.952+0000 7fa1cb0e5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: Active manager daemon x restarted 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: Activating manager daemon x 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: mgrmap e6: x(active, starting, since 0.0106108s) 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:33:17.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:33:17.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:33:17.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:33:17.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: Manager daemon x is now available 2024-04-25T18:33:17.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:17 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: Found migration_current of "None". Setting to last migration. 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:18.280 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:18 smithi063 ceph-mon[28006]: mgrmap e7: x(active, since 1.01839s) 2024-04-25T18:33:18.939 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:18.939 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T18:33:18.940 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T18:33:18.940 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:18.940 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.025+0000 7f493b400700 1 Processor -- start 2024-04-25T18:33:18.940 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.025+0000 7f493b400700 1 -- start start 2024-04-25T18:33:18.942 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.026+0000 7f493b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:18.942 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.026+0000 7f493b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4934063d60 con 0x7f4934108fb0 2024-04-25T18:33:18.942 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.026+0000 7f4938a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.026+0000 7f4938a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34010/0 (socket says 172.21.15.63:34010) 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.026+0000 7f4938a00700 1 -- 172.21.15.63:0/2962187080 learned_addr learned my addr 172.21.15.63:0/2962187080 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.027+0000 7f4938a00700 1 -- 172.21.15.63:0/2962187080 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f493410b8e0 con 0x7f4934108fb0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.027+0000 7f4938a00700 1 --2- 172.21.15.63:0/2962187080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49280098a0 tx=0x7f4928009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a85be29db1298b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.027+0000 7f4933400700 1 -- 172.21.15.63:0/2962187080 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4928004160 con 0x7f4934108fb0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.027+0000 7f4933400700 1 -- 172.21.15.63:0/2962187080 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4928004300 con 0x7f4934108fb0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.027+0000 7f4933400700 1 -- 172.21.15.63:0/2962187080 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4928011af0 con 0x7f4934108fb0 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 -- 172.21.15.63:0/2962187080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 msgr2=0x7f493410b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 --2- 172.21.15.63:0/2962187080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f49280098a0 tx=0x7f4928009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 -- 172.21.15.63:0/2962187080 shutdown_connections 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 --2- 172.21.15.63:0/2962187080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493410b3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.943 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 -- 172.21.15.63:0/2962187080 >> 172.21.15.63:0/2962187080 conn(0x7f4934102a40 msgr2=0x7f4934104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:18.944 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.028+0000 7f493b400700 1 -- 172.21.15.63:0/2962187080 shutdown_connections 2024-04-25T18:33:18.944 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.029+0000 7f493b400700 1 -- 172.21.15.63:0/2962187080 wait complete. 2024-04-25T18:33:18.944 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.030+0000 7f493b400700 1 Processor -- start 2024-04-25T18:33:18.944 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.030+0000 7f493b400700 1 -- start start 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.030+0000 7f493b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.030+0000 7f493b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f493410c8b0 con 0x7f4934108fb0 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.031+0000 7f4938a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.031+0000 7f4938a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34014/0 (socket says 172.21.15.63:34014) 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.031+0000 7f4938a00700 1 -- 172.21.15.63:0/1971606464 learned_addr learned my addr 172.21.15.63:0/1971606464 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.031+0000 7f4938a00700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f493419eb90 con 0x7f4934108fb0 2024-04-25T18:33:18.949 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.031+0000 7f4938a00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4928011ac0 tx=0x7f4928011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.032+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4928003b20 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.032+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f492801a8c0 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.032+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49280248d0 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.032+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f493419ee20 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.032+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49341a1b10 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4928022020 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4931600700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f491c03c1d0 con 0x7f491c039600 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f492804d570 con 0x7f4934108fb0 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4933e00700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.034+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:33:18.950 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.235+0000 7f4933e00700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.235+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.636+0000 7f4933e00700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:11.636+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:12.437+0000 7f4933e00700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:12.437+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:14.039+0000 7f4933e00700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/2210114679 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:14.039+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:16.963+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f492802d3f0 con 0x7f4934108fb0 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:16.963+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 msgr2=0x7f491c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:16.963+0000 7f4931600700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.966+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f492804ee20 con 0x7f4934108fb0 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.967+0000 7f4931600700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 0x7f491c03f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.967+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f492800f4c0 con 0x7f491c03cef0 2024-04-25T18:33:18.951 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.969+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 0x7f491c03f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.972+0000 7f4933e00700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 0x7f491c03f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4920003a10 tx=0x7f49200092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.973+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f492800f4c0 con 0x7f491c03cef0 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.986+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f4934069ba0 con 0x7f491c03cef0 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.987+0000 7f4931600700 1 -- 172.21.15.63:0/1971606464 <== mgr.14120 v2:172.21.15.63:6800/3403991527 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f4934069ba0 con 0x7f491c03cef0 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.987+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 msgr2=0x7f491c03f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.987+0000 7f493b400700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 0x7f491c03f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4920003a10 tx=0x7f49200092b0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.987+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 msgr2=0x7f493419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.987+0000 7f493b400700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4928011ac0 tx=0x7f4928011ba0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 shutdown_connections 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f491c03cef0 0x7f491c03f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4934108fb0 0x7f493419e650 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 --2- 172.21.15.63:0/1971606464 >> [v2:172.21.15.63:6800/2210114679,v1:172.21.15.63:6801/2210114679] conn(0x7f491c039600 0x7f491c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 >> 172.21.15.63:0/1971606464 conn(0x7f4934102a40 msgr2=0x7f4934103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 shutdown_connections 2024-04-25T18:33:18.952 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:17.988+0000 7f493b400700 1 -- 172.21.15.63:0/1971606464 wait complete. 2024-04-25T18:33:18.953 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 5 is available 2024-04-25T18:33:18.953 INFO:teuthology.orchestra.run.smithi063.stdout:Setting orchestrator backend to cephadm... 2024-04-25T18:33:19.292 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:18] ENGINE Bus STARTING 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:18] ENGINE Serving on http://172.21.15.63:8765 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:18] ENGINE Serving on https://172.21.15.63:7150 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:18] ENGINE Bus STARTED 2024-04-25T18:33:19.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:19 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:20 smithi063 ceph-mon[28006]: mgrmap e8: x(active, since 2s) 2024-04-25T18:33:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:20 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:20 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.599+0000 7f7e47c00700 1 Processor -- start 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.600+0000 7f7e47c00700 1 -- start start 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.600+0000 7f7e47c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.600+0000 7f7e47c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4010b150 con 0x7f7e4010a760 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.601+0000 7f7e45200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:20.642 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.601+0000 7f7e45200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42844/0 (socket says 172.21.15.63:42844) 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.601+0000 7f7e45200700 1 -- 172.21.15.63:0/2481700652 learned_addr learned my addr 172.21.15.63:0/2481700652 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.601+0000 7f7e45200700 1 -- 172.21.15.63:0/2481700652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4010b910 con 0x7f7e4010a760 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.602+0000 7f7e45200700 1 --2- 172.21.15.63:0/2481700652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e340098a0 tx=0x7f7e34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2261da580340cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.602+0000 7f7e3f400700 1 -- 172.21.15.63:0/2481700652 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e34004160 con 0x7f7e4010a760 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.602+0000 7f7e3f400700 1 -- 172.21.15.63:0/2481700652 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7e34004300 con 0x7f7e4010a760 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.602+0000 7f7e3f400700 1 -- 172.21.15.63:0/2481700652 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e34011af0 con 0x7f7e4010a760 2024-04-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 -- 172.21.15.63:0/2481700652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 msgr2=0x7f7e4010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2481700652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7e340098a0 tx=0x7f7e34009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 -- 172.21.15.63:0/2481700652 shutdown_connections 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2481700652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4010ab80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 -- 172.21.15.63:0/2481700652 >> 172.21.15.63:0/2481700652 conn(0x7f7e4007dc20 msgr2=0x7f7e401084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 -- 172.21.15.63:0/2481700652 shutdown_connections 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.603+0000 7f7e47c00700 1 -- 172.21.15.63:0/2481700652 wait complete. 2024-04-25T18:33:20.645 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.604+0000 7f7e47c00700 1 Processor -- start 2024-04-25T18:33:20.646 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.604+0000 7f7e47c00700 1 -- start start 2024-04-25T18:33:20.646 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e47c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:20.646 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e47c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4010ecc0 con 0x7f7e4010a760 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e45200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e45200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42846/0 (socket says 172.21.15.63:42846) 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e45200700 1 -- 172.21.15.63:0/2691689779 learned_addr learned my addr 172.21.15.63:0/2691689779 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.605+0000 7f7e45200700 1 -- 172.21.15.63:0/2691689779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4019ed30 con 0x7f7e4010a760 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e45200700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e340117b0 tx=0x7f7e34011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e34003af0 con 0x7f7e4010a760 2024-04-25T18:33:20.647 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7e34020070 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4019efc0 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e34019630 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.606+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4019fae0 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.607+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e400640a0 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.607+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f7e3401a9c0 con 0x7f7e4010a760 2024-04-25T18:33:20.648 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.608+0000 7f7e3d600700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 0x7f7e2803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:20.649 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.608+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7e3404dc00 con 0x7f7e4010a760 2024-04-25T18:33:20.649 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.608+0000 7f7e3fe00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 0x7f7e2803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:20.649 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.609+0000 7f7e3fe00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 0x7f7e2803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:20.649 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.615+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e3400e260 con 0x7f7e4010a760 2024-04-25T18:33:20.649 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.864+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f7e40108390 con 0x7f7e280395b0 2024-04-25T18:33:20.650 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.875+0000 7f7e3d600700 1 -- 172.21.15.63:0/2691689779 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7e40108390 con 0x7f7e280395b0 2024-04-25T18:33:20.650 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 msgr2=0x7f7e2803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:20.650 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 0x7f7e2803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c006fd0 tx=0x7f7e2c006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:20.650 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 msgr2=0x7f7e4019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:20.651 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7e340117b0 tx=0x7f7e34011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:20.651 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 shutdown_connections 2024-04-25T18:33:20.651 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f7e280395b0 0x7f7e2803ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:20.652 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 --2- 172.21.15.63:0/2691689779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4010a760 0x7f7e4019e7f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:20.652 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.878+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 >> 172.21.15.63:0/2691689779 conn(0x7f7e4007dc20 msgr2=0x7f7e40107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:20.652 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.879+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 shutdown_connections 2024-04-25T18:33:20.652 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:19.879+0000 7f7e47c00700 1 -- 172.21.15.63:0/2691689779 wait complete. 2024-04-25T18:33:21.225 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:21 smithi063 ceph-mon[28006]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:22.342 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:22 smithi063 ceph-mon[28006]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:22.350 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.291+0000 7f601ca00700 1 Processor -- start 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.291+0000 7f601ca00700 1 -- start start 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.292+0000 7f601ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.292+0000 7f601ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6018108f20 con 0x7f6018108530 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.292+0000 7f6015e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.293+0000 7f6015e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42856/0 (socket says 172.21.15.63:42856) 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.293+0000 7f6015e00700 1 -- 172.21.15.63:0/2500475629 learned_addr learned my addr 172.21.15.63:0/2500475629 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.293+0000 7f6015e00700 1 -- 172.21.15.63:0/2500475629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6018109730 con 0x7f6018108530 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.293+0000 7f6015e00700 1 --2- 172.21.15.63:0/2500475629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f60080098a0 tx=0x7f6008009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6f8fcc24d8b89a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:22.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.294+0000 7f6014a00700 1 -- 172.21.15.63:0/2500475629 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6008004160 con 0x7f6018108530 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.294+0000 7f6014a00700 1 -- 172.21.15.63:0/2500475629 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6008004300 con 0x7f6018108530 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.294+0000 7f6014a00700 1 -- 172.21.15.63:0/2500475629 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6008011af0 con 0x7f6018108530 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 -- 172.21.15.63:0/2500475629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 msgr2=0x7f6018108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 --2- 172.21.15.63:0/2500475629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f60080098a0 tx=0x7f6008009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 -- 172.21.15.63:0/2500475629 shutdown_connections 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 --2- 172.21.15.63:0/2500475629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f6018108950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 -- 172.21.15.63:0/2500475629 >> 172.21.15.63:0/2500475629 conn(0x7f6018103b50 msgr2=0x7f6018105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 -- 172.21.15.63:0/2500475629 shutdown_connections 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.295+0000 7f601ca00700 1 -- 172.21.15.63:0/2500475629 wait complete. 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.296+0000 7f601ca00700 1 Processor -- start 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.296+0000 7f601ca00700 1 -- start start 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.296+0000 7f601ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.296+0000 7f601ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f601810cae0 con 0x7f6018108530 2024-04-25T18:33:22.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.297+0000 7f6015e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.297+0000 7f6015e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42870/0 (socket says 172.21.15.63:42870) 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.297+0000 7f6015e00700 1 -- 172.21.15.63:0/4587275 learned_addr learned my addr 172.21.15.63:0/4587275 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.297+0000 7f6015e00700 1 -- 172.21.15.63:0/4587275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601819ec10 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.297+0000 7f6015e00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6008000c00 tx=0x7f6008011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.298+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6008003b20 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.298+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f600801a8c0 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.298+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6008024940 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.298+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601819eea0 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.298+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60181a1b90 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.299+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6018069ba0 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.300+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6008022020 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.300+0000 7f600ea00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 0x7f5ffc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.300+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f600804d890 con 0x7f6018108530 2024-04-25T18:33:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.301+0000 7f6015400700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 0x7f5ffc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.302+0000 7f6015400700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 0x7f5ffc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6004006fd0 tx=0x7f6004006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.306+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f600802c3a0 con 0x7f6018108530 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.541+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f6018104cc0 con 0x7f5ffc039600 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.542+0000 7f600ea00700 1 -- 172.21.15.63:0/4587275 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f6018104cc0 con 0x7f5ffc039600 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 msgr2=0x7f5ffc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 0x7f5ffc03bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6004006fd0 tx=0x7f6004006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 msgr2=0x7f601819e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6008000c00 tx=0x7f6008011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 shutdown_connections 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f5ffc039600 0x7f5ffc03bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 --2- 172.21.15.63:0/4587275 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6018108530 0x7f601819e6d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 >> 172.21.15.63:0/4587275 conn(0x7f6018103b50 msgr2=0x7f60181045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:22.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.545+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 shutdown_connections 2024-04-25T18:33:22.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:21.546+0000 7f601ca00700 1 -- 172.21.15.63:0/4587275 wait complete. 2024-04-25T18:33:22.360 INFO:teuthology.orchestra.run.smithi063.stdout:Generating ssh key... 2024-04-25T18:33:23.464 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: Generating public/private rsa key pair. 2024-04-25T18:33:23.464 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: Your identification has been saved in /tmp/tmpbhnvmq0w/key. 2024-04-25T18:33:23.464 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: Your public key has been saved in /tmp/tmpbhnvmq0w/key.pub. 2024-04-25T18:33:23.464 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: The key fingerprint is: 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: SHA256:VqJKiEWCFlroAAZA/1riIYpNhZnryZgAJjbi34llex4 ceph-e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: The key's randomart image is: 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: +---[RSA 3072]----+ 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |@=+ | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |*=.+ | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |O+=.. . . | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |B+.+. . o | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |oo+o.=. S | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |+B+oX.o. | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: |= ==.+ E | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: | o . | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: | . | 2024-04-25T18:33:23.465 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: +----[SHA256]-----+ 2024-04-25T18:33:24.214 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.966+0000 7f3cd5e00700 1 Processor -- start 2024-04-25T18:33:24.214 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.966+0000 7f3cd5e00700 1 -- start start 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.967+0000 7f3cd5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.967+0000 7f3cd5e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd010b170 con 0x7f3cd010a780 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.967+0000 7f3ccea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.967+0000 7f3ccea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42882/0 (socket says 172.21.15.63:42882) 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.967+0000 7f3ccea00700 1 -- 172.21.15.63:0/3040147499 learned_addr learned my addr 172.21.15.63:0/3040147499 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.968+0000 7f3ccea00700 1 -- 172.21.15.63:0/3040147499 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd010b930 con 0x7f3cd010a780 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.968+0000 7f3ccea00700 1 --2- 172.21.15.63:0/3040147499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00098a0 tx=0x7f3cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7964164d4e59dfaa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.968+0000 7f3ccd600700 1 -- 172.21.15.63:0/3040147499 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0004160 con 0x7f3cd010a780 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.968+0000 7f3ccd600700 1 -- 172.21.15.63:0/3040147499 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3cc0004300 con 0x7f3cd010a780 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.968+0000 7f3ccd600700 1 -- 172.21.15.63:0/3040147499 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0011af0 con 0x7f3cd010a780 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 -- 172.21.15.63:0/3040147499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 msgr2=0x7f3cd010aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/3040147499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00098a0 tx=0x7f3cc0009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:24.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 -- 172.21.15.63:0/3040147499 shutdown_connections 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/3040147499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd010aba0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 -- 172.21.15.63:0/3040147499 >> 172.21.15.63:0/3040147499 conn(0x7f3cd0105d40 msgr2=0x7f3cd0108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 -- 172.21.15.63:0/3040147499 shutdown_connections 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.969+0000 7f3cd5e00700 1 -- 172.21.15.63:0/3040147499 wait complete. 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.970+0000 7f3cd5e00700 1 Processor -- start 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.970+0000 7f3cd5e00700 1 -- start start 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3cd5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3cd5e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd010ece0 con 0x7f3cd010a780 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3ccea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3ccea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42884/0 (socket says 172.21.15.63:42884) 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3ccea00700 1 -- 172.21.15.63:0/1983941926 learned_addr learned my addr 172.21.15.63:0/1983941926 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.971+0000 7f3ccea00700 1 -- 172.21.15.63:0/1983941926 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd019ebc0 con 0x7f3cd010a780 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3ccea00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0000c00 tx=0x7f3cc0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:24.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0003b20 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3cc001a8c0 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0024940 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd019ee50 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.972+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd01a1b40 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.973+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd00640a0 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.973+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3cc0022020 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.974+0000 7f3cd4a00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 0x7f3cb403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.974+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3cc004d8a0 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.974+0000 7f3cce000700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 0x7f3cb403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.975+0000 7f3cce000700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 0x7f3cb403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc006fd0 tx=0x7f3cbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:22.980+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cc000be50 con 0x7f3cd010a780 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.209+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f3cd0106ef0 con 0x7f3cb4039600 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.441+0000 7f3cd4a00700 1 -- 172.21.15.63:0/1983941926 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3cd0106ef0 con 0x7f3cb4039600 2024-04-25T18:33:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.443+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 msgr2=0x7f3cb403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.443+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 0x7f3cb403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc006fd0 tx=0x7f3cbc006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.443+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 msgr2=0x7f3cd019e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.443+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0000c00 tx=0x7f3cc0011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 shutdown_connections 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f3cb4039600 0x7f3cb403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 --2- 172.21.15.63:0/1983941926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3cd010a780 0x7f3cd019e680 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 >> 172.21.15.63:0/1983941926 conn(0x7f3cd0105d40 msgr2=0x7f3cd01067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 shutdown_connections 2024-04-25T18:33:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:23.444+0000 7f3cd5e00700 1 -- 172.21.15.63:0/1983941926 wait complete. 2024-04-25T18:33:24.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:24 smithi063 ceph-mon[28006]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:24.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:24 smithi063 ceph-mon[28006]: Generating ssh key... 2024-04-25T18:33:24.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:24 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:24.520 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:24 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:25.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:33:25.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.832+0000 7f527e800700 1 Processor -- start 2024-04-25T18:33:25.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.832+0000 7f527e800700 1 -- start start 2024-04-25T18:33:25.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.833+0000 7f527e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:25.795 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.833+0000 7f527e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5278108f20 con 0x7f5278108530 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.833+0000 7f5277400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.833+0000 7f5277400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42898/0 (socket says 172.21.15.63:42898) 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.833+0000 7f5277400700 1 -- 172.21.15.63:0/3301852782 learned_addr learned my addr 172.21.15.63:0/3301852782 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.834+0000 7f5277400700 1 -- 172.21.15.63:0/3301852782 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5278109730 con 0x7f5278108530 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.834+0000 7f5277400700 1 --2- 172.21.15.63:0/3301852782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52680098a0 tx=0x7f5268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d1fbee309ee5e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.834+0000 7f5276a00700 1 -- 172.21.15.63:0/3301852782 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5268004160 con 0x7f5278108530 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.834+0000 7f5276a00700 1 -- 172.21.15.63:0/3301852782 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5268004300 con 0x7f5278108530 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.834+0000 7f5276a00700 1 -- 172.21.15.63:0/3301852782 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5268011af0 con 0x7f5278108530 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 -- 172.21.15.63:0/3301852782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 msgr2=0x7f5278108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 --2- 172.21.15.63:0/3301852782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f52680098a0 tx=0x7f5268009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 -- 172.21.15.63:0/3301852782 shutdown_connections 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 --2- 172.21.15.63:0/3301852782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f5278108950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 -- 172.21.15.63:0/3301852782 >> 172.21.15.63:0/3301852782 conn(0x7f5278103b50 msgr2=0x7f5278105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:25.796 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 -- 172.21.15.63:0/3301852782 shutdown_connections 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.835+0000 7f527e800700 1 -- 172.21.15.63:0/3301852782 wait complete. 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.836+0000 7f527e800700 1 Processor -- start 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.836+0000 7f527e800700 1 -- start start 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f527e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f527e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f527810cae0 con 0x7f5278108530 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f5277400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f5277400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42912/0 (socket says 172.21.15.63:42912) 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f5277400700 1 -- 172.21.15.63:0/3447145695 learned_addr learned my addr 172.21.15.63:0/3447145695 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.837+0000 7f5277400700 1 -- 172.21.15.63:0/3447145695 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527819c9c0 con 0x7f5278108530 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f5277400700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5268000c00 tx=0x7f5268011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5268003b20 con 0x7f5278108530 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f526801a8c0 con 0x7f5278108530 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f527819cc50 con 0x7f5278108530 2024-04-25T18:33:25.797 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5268024940 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.838+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f527819f940 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.839+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5268022020 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.839+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52780640a0 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.840+0000 7f5274c00700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 0x7f525c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.840+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f526804d7e0 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.840+0000 7f526fe00700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 0x7f525c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.841+0000 7f526fe00700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 0x7f525c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:24.846+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f526800be50 con 0x7f5278108530 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.080+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5278104d00 con 0x7f525c039600 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.081+0000 7f5274c00700 1 -- 172.21.15.63:0/3447145695 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5278104d00 con 0x7f525c039600 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 msgr2=0x7f525c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 0x7f525c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5260006fd0 tx=0x7f5260006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:25.798 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 msgr2=0x7f527819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5268000c00 tx=0x7f5268011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 shutdown_connections 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f525c039600 0x7f525c03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 --2- 172.21.15.63:0/3447145695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5278108530 0x7f527819c480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.084+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 >> 172.21.15.63:0/3447145695 conn(0x7f5278103b50 msgr2=0x7f52781045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.085+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 shutdown_connections 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:25.085+0000 7f527e800700 1 -- 172.21.15.63:0/3447145695 wait complete. 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T18:33:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T18:33:25.800 INFO:teuthology.orchestra.run.smithi063.stdout:Adding host smithi063... 2024-04-25T18:33:26.106 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:25 smithi063 ceph-mon[28006]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:26.958 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:26 smithi063 ceph-mon[28006]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:29.040 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:28 smithi063 ceph-mon[28006]: Deploying cephadm binary to smithi063 2024-04-25T18:33:33.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Added host 'smithi063' with addr '172.21.15.63' 2024-04-25T18:33:33.955 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.455+0000 7f9c92e00700 1 Processor -- start 2024-04-25T18:33:33.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.456+0000 7f9c92e00700 1 -- start start 2024-04-25T18:33:33.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.456+0000 7f9c92e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:33.956 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.456+0000 7f9c92e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c10af50 con 0x7f9c8c10a560 2024-04-25T18:33:33.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.456+0000 7f9c91a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:33.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.457+0000 7f9c91a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42916/0 (socket says 172.21.15.63:42916) 2024-04-25T18:33:33.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.457+0000 7f9c91a00700 1 -- 172.21.15.63:0/3327424785 learned_addr learned my addr 172.21.15.63:0/3327424785 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:33.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.457+0000 7f9c91a00700 1 -- 172.21.15.63:0/3327424785 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8c10b710 con 0x7f9c8c10a560 2024-04-25T18:33:33.957 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.457+0000 7f9c91a00700 1 --2- 172.21.15.63:0/3327424785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c800098a0 tx=0x7f9c80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398b62a075c36238 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:33.958 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c8be00700 1 -- 172.21.15.63:0/3327424785 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80004160 con 0x7f9c8c10a560 2024-04-25T18:33:33.958 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c8be00700 1 -- 172.21.15.63:0/3327424785 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9c80004300 con 0x7f9c8c10a560 2024-04-25T18:33:33.958 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c8be00700 1 -- 172.21.15.63:0/3327424785 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80011af0 con 0x7f9c8c10a560 2024-04-25T18:33:33.958 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c92e00700 1 -- 172.21.15.63:0/3327424785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 msgr2=0x7f9c8c10a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:33.959 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c92e00700 1 --2- 172.21.15.63:0/3327424785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9c800098a0 tx=0x7f9c80009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:33.959 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.458+0000 7f9c92e00700 1 -- 172.21.15.63:0/3327424785 shutdown_connections 2024-04-25T18:33:33.959 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.459+0000 7f9c92e00700 1 --2- 172.21.15.63:0/3327424785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c10a980 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:33.959 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.459+0000 7f9c92e00700 1 -- 172.21.15.63:0/3327424785 >> 172.21.15.63:0/3327424785 conn(0x7f9c8c105b80 msgr2=0x7f9c8c107fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:33.959 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.459+0000 7f9c92e00700 1 -- 172.21.15.63:0/3327424785 shutdown_connections 2024-04-25T18:33:33.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.459+0000 7f9c92e00700 1 -- 172.21.15.63:0/3327424785 wait complete. 2024-04-25T18:33:33.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.460+0000 7f9c92e00700 1 Processor -- start 2024-04-25T18:33:33.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.460+0000 7f9c92e00700 1 -- start start 2024-04-25T18:33:33.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.460+0000 7f9c92e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:33.960 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.460+0000 7f9c92e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8c10eac0 con 0x7f9c8c10a560 2024-04-25T18:33:33.961 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c91a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:33.961 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c91a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42918/0 (socket says 172.21.15.63:42918) 2024-04-25T18:33:33.961 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c91a00700 1 -- 172.21.15.63:0/512075048 learned_addr learned my addr 172.21.15.63:0/512075048 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:33.961 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c91a00700 1 -- 172.21.15.63:0/512075048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8c19e9c0 con 0x7f9c8c10a560 2024-04-25T18:33:33.962 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c91a00700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9c80000c00 tx=0x7f9c80011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:33.962 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.461+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80003b20 con 0x7f9c8c10a560 2024-04-25T18:33:33.963 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.462+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9c8001a8c0 con 0x7f9c8c10a560 2024-04-25T18:33:33.963 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.462+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8c19ec50 con 0x7f9c8c10a560 2024-04-25T18:33:33.963 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.462+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c80024940 con 0x7f9c8c10a560 2024-04-25T18:33:33.963 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.462+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8c1a1940 con 0x7f9c8c10a560 2024-04-25T18:33:33.963 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.463+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c8c0640a0 con 0x7f9c8c10a560 2024-04-25T18:33:33.965 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.463+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9c80022020 con 0x7f9c8c10a560 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.463+0000 7f9c8a000700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 0x7f9c7403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.463+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9c8004d7e0 con 0x7f9c8c10a560 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.464+0000 7f9c91000700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 0x7f9c7403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.464+0000 7f9c91000700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 0x7f9c7403bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006fd0 tx=0x7f9c78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.469+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c8002c3a0 con 0x7f9c8c10a560 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:26.702+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}) -- 0x7f9c8c106dc0 con 0x7f9c74039600 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.146+0000 7f9c8a000700 1 -- 172.21.15.63:0/512075048 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9c8c106dc0 con 0x7f9c74039600 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.150+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 msgr2=0x7f9c7403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.150+0000 7f9c92e00700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 0x7f9c7403bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9c78006fd0 tx=0x7f9c78006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 msgr2=0x7f9c8c19e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9c80000c00 tx=0x7f9c80011b70 comp rx=0 tx=0).stop 2024-04-25T18:33:33.966 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 shutdown_connections 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f9c74039600 0x7f9c7403bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 --2- 172.21.15.63:0/512075048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c8c10a560 0x7f9c8c19e480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 >> 172.21.15.63:0/512075048 conn(0x7f9c8c105b80 msgr2=0x7f9c8c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 shutdown_connections 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:33.151+0000 7f9c92e00700 1 -- 172.21.15.63:0/512075048 wait complete. 2024-04-25T18:33:33.967 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying unmanaged mon service... 2024-04-25T18:33:34.313 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:34 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:34.313 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:34 smithi063 ceph-mon[28006]: Added host smithi063 2024-04-25T18:33:34.313 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:34 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:36.319 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:35 smithi063 ceph-mon[28006]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:36.319 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:35 smithi063 ceph-mon[28006]: Saving service mon spec with placement count:5 2024-04-25T18:33:36.319 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:35 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.660+0000 7ff760600700 1 Processor -- start 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.660+0000 7ff760600700 1 -- start start 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.661+0000 7ff760600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.661+0000 7ff760600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff758108ef0 con 0x7ff758108500 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.661+0000 7ff75dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.661+0000 7ff75dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46100/0 (socket says 172.21.15.63:46100) 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.661+0000 7ff75dc00700 1 -- 172.21.15.63:0/2873505611 learned_addr learned my addr 172.21.15.63:0/2873505611 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.662+0000 7ff75dc00700 1 -- 172.21.15.63:0/2873505611 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff758109700 con 0x7ff758108500 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.662+0000 7ff75dc00700 1 --2- 172.21.15.63:0/2873505611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009e80 tx=0x7ff74c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a306baed91c174a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.662+0000 7ff757e00700 1 -- 172.21.15.63:0/2873505611 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff74c017070 con 0x7ff758108500 2024-04-25T18:33:36.333 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.662+0000 7ff757e00700 1 -- 172.21.15.63:0/2873505611 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff74c00b8c0 con 0x7ff758108500 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.662+0000 7ff757e00700 1 -- 172.21.15.63:0/2873505611 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff74c004490 con 0x7ff758108500 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 -- 172.21.15.63:0/2873505611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 msgr2=0x7ff758108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 --2- 172.21.15.63:0/2873505611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009e80 tx=0x7ff74c00b330 comp rx=0 tx=0).stop 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 -- 172.21.15.63:0/2873505611 shutdown_connections 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 --2- 172.21.15.63:0/2873505611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff758108920 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 -- 172.21.15.63:0/2873505611 >> 172.21.15.63:0/2873505611 conn(0x7ff758103b00 msgr2=0x7ff758105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 -- 172.21.15.63:0/2873505611 shutdown_connections 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.663+0000 7ff760600700 1 -- 172.21.15.63:0/2873505611 wait complete. 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.664+0000 7ff760600700 1 Processor -- start 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.664+0000 7ff760600700 1 -- start start 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff760600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff760600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff75810cab0 con 0x7ff758108500 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff75dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:36.334 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff75dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46108/0 (socket says 172.21.15.63:46108) 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff75dc00700 1 -- 172.21.15.63:0/3440475078 learned_addr learned my addr 172.21.15.63:0/3440475078 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff75dc00700 1 -- 172.21.15.63:0/3440475078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75819ebe0 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.665+0000 7ff75dc00700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00b7b0 tx=0x7ff74c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.666+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff74c020020 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.666+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff74c012430 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.666+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75819ee70 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.666+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff74c017040 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.666+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7581a1b60 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.667+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7580640a0 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.667+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ff74c023020 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.668+0000 7ff756000700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 0x7ff74003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.668+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff74c00f050 con 0x7ff758108500 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.668+0000 7ff75d200700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 0x7ff74003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:36.335 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.669+0000 7ff75d200700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 0x7ff74003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff744006fd0 tx=0x7ff744006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.673+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff74c02a020 con 0x7ff758108500 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.951+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7ff758104d20 con 0x7ff7400395b0 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.962+0000 7ff756000700 1 -- 172.21.15.63:0/3440475078 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff758104d20 con 0x7ff7400395b0 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.965+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 msgr2=0x7ff74003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.965+0000 7ff760600700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 0x7ff74003ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff744006fd0 tx=0x7ff744006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.965+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 msgr2=0x7ff75819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.965+0000 7ff760600700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff74c00b7b0 tx=0x7ff74c00be20 comp rx=0 tx=0).stop 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 shutdown_connections 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7ff7400395b0 0x7ff74003ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 --2- 172.21.15.63:0/3440475078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff758108500 0x7ff75819e6a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 >> 172.21.15.63:0/3440475078 conn(0x7ff758103b00 msgr2=0x7ff758104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 shutdown_connections 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:34.966+0000 7ff760600700 1 -- 172.21.15.63:0/3440475078 wait complete. 2024-04-25T18:33:36.336 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying unmanaged mgr service... 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: Saving service mgr spec with placement count:2 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:38.058 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:37 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:38.354 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T18:33:38.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.406+0000 7f01e8e00700 1 Processor -- start 2024-04-25T18:33:38.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.407+0000 7f01e8e00700 1 -- start start 2024-04-25T18:33:38.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.407+0000 7f01e8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:38.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.407+0000 7f01e8e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f01e410af80 con 0x7f01e410a590 2024-04-25T18:33:38.355 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.408+0000 7f01e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.408+0000 7f01e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49782/0 (socket says 172.21.15.63:49782) 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.408+0000 7f01e3400700 1 -- 172.21.15.63:0/3947669798 learned_addr learned my addr 172.21.15.63:0/3947669798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.408+0000 7f01e3400700 1 -- 172.21.15.63:0/3947669798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e410b740 con 0x7f01e410a590 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e3400700 1 --2- 172.21.15.63:0/3947669798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01d40098a0 tx=0x7f01d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1349b17ec921a126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e2a00700 1 -- 172.21.15.63:0/3947669798 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d4004160 con 0x7f01e410a590 2024-04-25T18:33:38.356 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e2a00700 1 -- 172.21.15.63:0/3947669798 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d4004300 con 0x7f01e410a590 2024-04-25T18:33:38.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e2a00700 1 -- 172.21.15.63:0/3947669798 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d4011a20 con 0x7f01e410a590 2024-04-25T18:33:38.357 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e8e00700 1 -- 172.21.15.63:0/3947669798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 msgr2=0x7f01e410a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.409+0000 7f01e8e00700 1 --2- 172.21.15.63:0/3947669798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01d40098a0 tx=0x7f01d4009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.410+0000 7f01e8e00700 1 -- 172.21.15.63:0/3947669798 shutdown_connections 2024-04-25T18:33:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.410+0000 7f01e8e00700 1 --2- 172.21.15.63:0/3947669798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e410a9b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.410+0000 7f01e8e00700 1 -- 172.21.15.63:0/3947669798 >> 172.21.15.63:0/3947669798 conn(0x7f01e407da00 msgr2=0x7f01e41084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.410+0000 7f01e8e00700 1 -- 172.21.15.63:0/3947669798 shutdown_connections 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.410+0000 7f01e8e00700 1 -- 172.21.15.63:0/3947669798 wait complete. 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.411+0000 7f01e8e00700 1 Processor -- start 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.411+0000 7f01e8e00700 1 -- start start 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.411+0000 7f01e8e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.411+0000 7f01e8e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f01e410eaf0 con 0x7f01e410a590 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.412+0000 7f01e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:38.359 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.412+0000 7f01e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49792/0 (socket says 172.21.15.63:49792) 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.412+0000 7f01e3400700 1 -- 172.21.15.63:0/2798841011 learned_addr learned my addr 172.21.15.63:0/2798841011 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.412+0000 7f01e3400700 1 -- 172.21.15.63:0/2798841011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e419e9d0 con 0x7f01e410a590 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.412+0000 7f01e3400700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f01d4000c00 tx=0x7f01d4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.413+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d4003aa0 con 0x7f01e410a590 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.413+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01d401b8f0 con 0x7f01e410a590 2024-04-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.413+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e419ec60 con 0x7f01e410a590 2024-04-25T18:33:38.361 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.413+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01d4023890 con 0x7f01e410a590 2024-04-25T18:33:38.361 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.413+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e41a1950 con 0x7f01e410a590 2024-04-25T18:33:38.361 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.414+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e4069ba0 con 0x7f01e410a590 2024-04-25T18:33:38.361 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.415+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f01d401b440 con 0x7f01e410a590 2024-04-25T18:33:38.361 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.415+0000 7f01e0c00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 0x7f01c803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:38.362 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.415+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f01d404de40 con 0x7f01e410a590 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.415+0000 7f01daa00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 0x7f01c803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.416+0000 7f01daa00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 0x7f01c803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01cc006fd0 tx=0x7f01cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.421+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01d401f030 con 0x7f01e410a590 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.672+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f01e4108330 con 0x7f01c8039600 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.679+0000 7f01e0c00700 1 -- 172.21.15.63:0/2798841011 <== mgr.14120 v2:172.21.15.63:6800/3403991527 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f01e4108330 con 0x7f01c8039600 2024-04-25T18:33:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 msgr2=0x7f01c803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 0x7f01c803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01cc006fd0 tx=0x7f01cc006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 msgr2=0x7f01e419e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f01d4000c00 tx=0x7f01d4011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 shutdown_connections 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f01c8039600 0x7f01c803bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 --2- 172.21.15.63:0/2798841011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f01e410a590 0x7f01e419e490 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.683+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 >> 172.21.15.63:0/2798841011 conn(0x7f01e407da00 msgr2=0x7f01e4107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:38.365 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.684+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 shutdown_connections 2024-04-25T18:33:38.365 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:37.684+0000 7f01e8e00700 1 -- 172.21.15.63:0/2798841011 wait complete. 2024-04-25T18:33:40.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:40 smithi063 ceph-mon[28006]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:33:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:40 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/128331302' entity='client.admin' 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.061+0000 7fa45a200700 1 Processor -- start 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa45a200700 1 -- start start 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa45a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa45a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa45407be40 con 0x7fa454109040 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa453400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa453400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49808/0 (socket says 172.21.15.63:49808) 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.062+0000 7fa453400700 1 -- 172.21.15.63:0/259015361 learned_addr learned my addr 172.21.15.63:0/259015361 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.063+0000 7fa453400700 1 -- 172.21.15.63:0/259015361 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45410b970 con 0x7fa454109040 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.063+0000 7fa453400700 1 --2- 172.21.15.63:0/259015361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa444009e80 tx=0x7fa44400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3cd36475c24796 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.063+0000 7fa452000700 1 -- 172.21.15.63:0/259015361 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444017070 con 0x7fa454109040 2024-04-25T18:33:40.684 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa452000700 1 -- 172.21.15.63:0/259015361 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa44400b8c0 con 0x7fa454109040 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa452000700 1 -- 172.21.15.63:0/259015361 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444004490 con 0x7fa454109040 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa45a200700 1 -- 172.21.15.63:0/259015361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 msgr2=0x7fa45410b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa45a200700 1 --2- 172.21.15.63:0/259015361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa444009e80 tx=0x7fa44400b330 comp rx=0 tx=0).stop 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa45a200700 1 -- 172.21.15.63:0/259015361 shutdown_connections 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa45a200700 1 --2- 172.21.15.63:0/259015361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45410b430 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.064+0000 7fa45a200700 1 -- 172.21.15.63:0/259015361 >> 172.21.15.63:0/259015361 conn(0x7fa454102a50 msgr2=0x7fa454104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.065+0000 7fa45a200700 1 -- 172.21.15.63:0/259015361 shutdown_connections 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.065+0000 7fa45a200700 1 -- 172.21.15.63:0/259015361 wait complete. 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa45a200700 1 Processor -- start 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa45a200700 1 -- start start 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa45a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa45a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa45410c910 con 0x7fa454109040 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa453400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:40.685 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa453400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49810/0 (socket says 172.21.15.63:49810) 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.066+0000 7fa453400700 1 -- 172.21.15.63:0/128331302 learned_addr learned my addr 172.21.15.63:0/128331302 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.067+0000 7fa453400700 1 -- 172.21.15.63:0/128331302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45419ebd0 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.067+0000 7fa453400700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa44400b7b0 tx=0x7fa44400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.067+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444020020 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.067+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa444003c60 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.067+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444017040 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.068+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45419ee60 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.068+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4541a1b50 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.069+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa444023020 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.069+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4540640a0 con 0x7fa454109040 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.069+0000 7fa44be00700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 0x7fa43803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:40.686 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.070+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa44400f050 con 0x7fa454109040 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.070+0000 7fa452a00700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 0x7fa43803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.071+0000 7fa452a00700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 0x7fa43803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa440006fd0 tx=0x7fa440006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.076+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa44402a020 con 0x7fa454109040 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.318+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa4541a2120 con 0x7fa454109040 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.324+0000 7fa44be00700 1 -- 172.21.15.63:0/128331302 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa44402c300 con 0x7fa454109040 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.326+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 msgr2=0x7fa43803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.326+0000 7fa45a200700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 0x7fa43803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa440006fd0 tx=0x7fa440006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.326+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 msgr2=0x7fa45419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.326+0000 7fa45a200700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa44400b7b0 tx=0x7fa44400bd50 comp rx=0 tx=0).stop 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 shutdown_connections 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7fa438039600 0x7fa43803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 --2- 172.21.15.63:0/128331302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa454109040 0x7fa45419e690 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 >> 172.21.15.63:0/128331302 conn(0x7fa454102a50 msgr2=0x7fa454103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:40.687 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 shutdown_connections 2024-04-25T18:33:40.688 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:39.327+0000 7fa45a200700 1 -- 172.21.15.63:0/128331302 wait complete. 2024-04-25T18:33:41.581 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:41 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.330+0000 7f30abc00700 1 Processor -- start 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.330+0000 7f30abc00700 1 -- start start 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.331+0000 7f30abc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.331+0000 7f30abc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f30a4108f20 con 0x7f30a4108530 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.331+0000 7f30a9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.331+0000 7f30a9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49824/0 (socket says 172.21.15.63:49824) 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.331+0000 7f30a9200700 1 -- 172.21.15.63:0/2042881671 learned_addr learned my addr 172.21.15.63:0/2042881671 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.332+0000 7f30a9200700 1 -- 172.21.15.63:0/2042881671 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a4109730 con 0x7f30a4108530 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.332+0000 7f30a9200700 1 --2- 172.21.15.63:0/2042881671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f30980098a0 tx=0x7f3098009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a345295797fc6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.332+0000 7f30a3400700 1 -- 172.21.15.63:0/2042881671 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3098004160 con 0x7f30a4108530 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.332+0000 7f30a3400700 1 -- 172.21.15.63:0/2042881671 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3098004300 con 0x7f30a4108530 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.332+0000 7f30a3400700 1 -- 172.21.15.63:0/2042881671 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3098011a20 con 0x7f30a4108530 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 -- 172.21.15.63:0/2042881671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 msgr2=0x7f30a4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 --2- 172.21.15.63:0/2042881671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f30980098a0 tx=0x7f3098009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 -- 172.21.15.63:0/2042881671 shutdown_connections 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 --2- 172.21.15.63:0/2042881671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a4108950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 -- 172.21.15.63:0/2042881671 >> 172.21.15.63:0/2042881671 conn(0x7f30a4103b50 msgr2=0x7f30a4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 -- 172.21.15.63:0/2042881671 shutdown_connections 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.333+0000 7f30abc00700 1 -- 172.21.15.63:0/2042881671 wait complete. 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.334+0000 7f30abc00700 1 Processor -- start 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30abc00700 1 -- start start 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30abc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30abc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f30a410cae0 con 0x7f30a4108530 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30a9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:42.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30a9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49838/0 (socket says 172.21.15.63:49838) 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.335+0000 7f30a9200700 1 -- 172.21.15.63:0/4199333202 learned_addr learned my addr 172.21.15.63:0/4199333202 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30a9200700 1 -- 172.21.15.63:0/4199333202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a419c940 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30a9200700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30980116e0 tx=0x7f3098011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30980039e0 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309801b8f0 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a419cbd0 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3098023760 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.336+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a419f8c0 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.337+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f309801b440 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.337+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a40640a0 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.338+0000 7f30a1600700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 0x7f308c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.338+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f309804df40 con 0x7f30a4108530 2024-04-25T18:33:42.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.338+0000 7f30a3e00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 0x7f308c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.339+0000 7f30a3e00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 0x7f308c03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3090006fd0 tx=0x7f3090006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.344+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f309801f030 con 0x7f30a4108530 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.579+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f30a419fee0 con 0x7f30a4108530 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.584+0000 7f30a1600700 1 -- 172.21.15.63:0/4199333202 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f309801bd50 con 0x7f30a4108530 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 msgr2=0x7f308c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 0x7f308c03ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3090006fd0 tx=0x7f3090006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 msgr2=0x7f30a419c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f30980116e0 tx=0x7f3098011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 shutdown_connections 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f308c0395b0 0x7f308c03ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:42.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 --2- 172.21.15.63:0/4199333202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30a4108530 0x7f30a419c400 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:42.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 >> 172.21.15.63:0/4199333202 conn(0x7f30a4103b50 msgr2=0x7f30a41045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:42.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 shutdown_connections 2024-04-25T18:33:42.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:41.587+0000 7f30abc00700 1 -- 172.21.15.63:0/4199333202 wait complete. 2024-04-25T18:33:42.380 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling the dashboard module... 2024-04-25T18:33:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:42 smithi063 ceph-mon[28006]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:33:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:42 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4199333202' entity='client.admin' 2024-04-25T18:33:43.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:43 smithi063 ceph-mon[28006]: from='mgr.14120 172.21.15.63:0/4087687150' entity='mgr.x' 2024-04-25T18:33:43.872 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:43 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2736811523' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T18:33:43.873 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setuser ceph since I am not root 2024-04-25T18:33:43.873 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setgroup ceph since I am not root 2024-04-25T18:33:44.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:44.022+0000 7f1dd10c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:33:44.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:44.271+0000 7f1dd10c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:33:44.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:44.406+0000 7f1dd10c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:33:44.934 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:44.816+0000 7f1dd10c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:33:44.935 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2736811523' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T18:33:44.935 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-mon[28006]: mgrmap e9: x(active, since 26s) 2024-04-25T18:33:45.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.969+0000 7f9495800700 1 Processor -- start 2024-04-25T18:33:45.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.970+0000 7f9495800700 1 -- start start 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.970+0000 7f9495800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.970+0000 7f9495800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9490063d60 con 0x7f9490108dd0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.970+0000 7f948fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.971+0000 7f948fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49846/0 (socket says 172.21.15.63:49846) 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.971+0000 7f948fe00700 1 -- 172.21.15.63:0/1210964881 learned_addr learned my addr 172.21.15.63:0/1210964881 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.971+0000 7f948fe00700 1 -- 172.21.15.63:0/1210964881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949010b700 con 0x7f9490108dd0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.971+0000 7f948fe00700 1 --2- 172.21.15.63:0/1210964881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94800098a0 tx=0x7f9480009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf8bce7f494b3c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.972+0000 7f948ea00700 1 -- 172.21.15.63:0/1210964881 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9480004160 con 0x7f9490108dd0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.972+0000 7f948ea00700 1 -- 172.21.15.63:0/1210964881 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9480004300 con 0x7f9490108dd0 2024-04-25T18:33:45.005 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.972+0000 7f948ea00700 1 -- 172.21.15.63:0/1210964881 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9480011a20 con 0x7f9490108dd0 2024-04-25T18:33:45.009 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 -- 172.21.15.63:0/1210964881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 msgr2=0x7f949010b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:45.009 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 --2- 172.21.15.63:0/1210964881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94800098a0 tx=0x7f9480009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:45.009 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 -- 172.21.15.63:0/1210964881 shutdown_connections 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 --2- 172.21.15.63:0/1210964881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949010b1c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 -- 172.21.15.63:0/1210964881 >> 172.21.15.63:0/1210964881 conn(0x7f9490102820 msgr2=0x7f9490104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 -- 172.21.15.63:0/1210964881 shutdown_connections 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.973+0000 7f9495800700 1 -- 172.21.15.63:0/1210964881 wait complete. 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.974+0000 7f9495800700 1 Processor -- start 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.974+0000 7f9495800700 1 -- start start 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f9495800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f9495800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f949010c710 con 0x7f9490108dd0 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f948fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f948fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49860/0 (socket says 172.21.15.63:49860) 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f948fe00700 1 -- 172.21.15.63:0/2736811523 learned_addr learned my addr 172.21.15.63:0/2736811523 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.975+0000 7f948fe00700 1 -- 172.21.15.63:0/2736811523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949019ea10 con 0x7f9490108dd0 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.976+0000 7f948fe00700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9480004660 tx=0x7f9480011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.976+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9480003aa0 con 0x7f9490108dd0 2024-04-25T18:33:45.010 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.976+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f948001b8f0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.976+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9480023900 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.976+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949019eca0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.977+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94901a1990 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.978+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f948001baa0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.978+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9490069ba0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.978+0000 7f948cc00700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 0x7f947403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.979+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f948004df70 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.979+0000 7f948f400700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 0x7f947403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.980+0000 7f948f400700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 0x7f947403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f947c006fd0 tx=0x7f947c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:42.984+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f948001f030 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.278+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f94901a1f80 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.710+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f948001a3d0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.714+0000 7f948cc00700 1 -- 172.21.15.63:0/2736811523 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f9480016ca0 con 0x7f9490108dd0 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.714+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 msgr2=0x7f947403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.714+0000 7f9495800700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 0x7f947403ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f947c006fd0 tx=0x7f947c006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:45.011 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.714+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 msgr2=0x7f949019e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.714+0000 7f9495800700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9480004660 tx=0x7f9480011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.715+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 shutdown_connections 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.715+0000 7f9495800700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f94740395b0 0x7f947403ba70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.715+0000 7f9495800700 1 --2- 172.21.15.63:0/2736811523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490108dd0 0x7f949019e4d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.715+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 >> 172.21.15.63:0/2736811523 conn(0x7f9490102820 msgr2=0x7f94901032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.717+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 shutdown_connections 2024-04-25T18:33:45.012 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:43.717+0000 7f9495800700 1 -- 172.21.15.63:0/2736811523 wait complete. 2024-04-25T18:33:45.186 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:44.932+0000 7f1dd10c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:33:46.772 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:46.586+0000 7f1dd10c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:33:46.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:46 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4000667936' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T18:33:47.047 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:46.824+0000 7f1dd10c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:33:47.047 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:47.044+0000 7f1dd10c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:33:47.088 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:47.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T18:33:47.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T18:33:47.089 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.074+0000 7f1c73400700 1 Processor -- start 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.075+0000 7f1c73400700 1 -- start start 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.075+0000 7f1c73400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.075+0000 7f1c73400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c063d40 con 0x7f1c6c108fd0 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.076+0000 7f1c70a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:47.094 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.076+0000 7f1c70a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49890/0 (socket says 172.21.15.63:49890) 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.076+0000 7f1c70a00700 1 -- 172.21.15.63:0/1540325404 learned_addr learned my addr 172.21.15.63:0/1540325404 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.076+0000 7f1c70a00700 1 -- 172.21.15.63:0/1540325404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c10b900 con 0x7f1c6c108fd0 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.077+0000 7f1c70a00700 1 --2- 172.21.15.63:0/1540325404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c600098a0 tx=0x7f1c60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a996830c0a93c09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.077+0000 7f1c6b400700 1 -- 172.21.15.63:0/1540325404 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60004160 con 0x7f1c6c108fd0 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.077+0000 7f1c6b400700 1 -- 172.21.15.63:0/1540325404 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c60004300 con 0x7f1c6c108fd0 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.077+0000 7f1c6b400700 1 -- 172.21.15.63:0/1540325404 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60011a20 con 0x7f1c6c108fd0 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 -- 172.21.15.63:0/1540325404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 msgr2=0x7f1c6c10b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 --2- 172.21.15.63:0/1540325404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1c600098a0 tx=0x7f1c60009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 -- 172.21.15.63:0/1540325404 shutdown_connections 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 --2- 172.21.15.63:0/1540325404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c10b3c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 -- 172.21.15.63:0/1540325404 >> 172.21.15.63:0/1540325404 conn(0x7f1c6c102a40 msgr2=0x7f1c6c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 -- 172.21.15.63:0/1540325404 shutdown_connections 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.078+0000 7f1c73400700 1 -- 172.21.15.63:0/1540325404 wait complete. 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.080+0000 7f1c73400700 1 Processor -- start 2024-04-25T18:33:47.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.080+0000 7f1c73400700 1 -- start start 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.080+0000 7f1c73400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.080+0000 7f1c73400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6c10c8a0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.081+0000 7f1c70a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.081+0000 7f1c70a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49902/0 (socket says 172.21.15.63:49902) 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.081+0000 7f1c70a00700 1 -- 172.21.15.63:0/4000667936 learned_addr learned my addr 172.21.15.63:0/4000667936 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.081+0000 7f1c70a00700 1 -- 172.21.15.63:0/4000667936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c6c19eb10 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.081+0000 7f1c70a00700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c60000c00 tx=0x7f1c60011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.082+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60003aa0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.082+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c6001b8f0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.082+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1c60023870 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.082+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c6c19ee00 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.082+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c6c1a1a90 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.083+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f1c6001b440 con 0x7f1c6c108fd0 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.083+0000 7f1c69600700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 0x7f1c5403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:47.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.083+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c6c0640a0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.083+0000 7f1c6be00700 1 -- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 msgr2=0x7f1c5403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:47.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.084+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1c6004e350 con 0x7f1c6c108fd0 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.084+0000 7f1c6be00700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 0x7f1c5403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.090+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c6001f030 con 0x7f1c6c108fd0 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.285+0000 7f1c6be00700 1 -- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 msgr2=0x7f1c5403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.285+0000 7f1c6be00700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 0x7f1c5403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.405+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f1c6c1a21f0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.406+0000 7f1c69600700 1 -- 172.21.15.63:0/4000667936 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f1c60023ad0 con 0x7f1c6c108fd0 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.408+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 msgr2=0x7f1c5403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.408+0000 7f1c73400700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 0x7f1c5403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.408+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 msgr2=0x7f1c6c19e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.408+0000 7f1c73400700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c60000c00 tx=0x7f1c60011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 shutdown_connections 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f1c54039660 0x7f1c5403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 --2- 172.21.15.63:0/4000667936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c6c108fd0 0x7f1c6c19e5d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:47.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 >> 172.21.15.63:0/4000667936 conn(0x7f1c6c102a40 msgr2=0x7f1c6c103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:47.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 shutdown_connections 2024-04-25T18:33:47.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:46.409+0000 7f1c73400700 1 -- 172.21.15.63:0/4000667936 wait complete. 2024-04-25T18:33:47.100 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-04-25T18:33:47.100 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 9... 2024-04-25T18:33:47.680 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:47.446+0000 7f1dd10c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:33:47.681 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:47.572+0000 7f1dd10c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:33:47.962 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:47.834+0000 7f1dd10c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:33:48.363 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:48.096+0000 7f1dd10c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:33:49.164 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:48.903+0000 7f1dd10c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:33:49.876 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:49.476+0000 7f1dd10c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:33:49.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:49.610+0000 7f1dd10c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:33:50.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:49.892+0000 7f1dd10c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:33:50.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:50.569+0000 7f1dd10c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:33:50.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:50.697+0000 7f1dd10c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:33:51.349 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:51.033+0000 7f1dd10c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:33:51.349 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:51.216+0000 7f1dd10c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:33:51.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:51.347+0000 7f1dd10c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:33:51.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:51.503+0000 7f1dd10c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:33:52.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:51.877+0000 7f1dd10c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:33:52.957 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:52.662+0000 7f1dd10c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:33:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:33:52.955+0000 7f1dd10c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: Active manager daemon x restarted 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: Activating manager daemon x 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: mgrmap e10: x(active, starting, since 0.010913s) 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:33:53.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:33:53.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:33:53.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:33:53.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:53 smithi063 ceph-mon[28006]: Manager daemon x is now available 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:53] ENGINE Bus STARTING 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:53] ENGINE Serving on http://172.21.15.63:8765 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:53] ENGINE Serving on https://172.21.15.63:7150 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: [25/Apr/2024:18:33:53] ENGINE Bus STARTED 2024-04-25T18:33:54.339 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:54 smithi063 ceph-mon[28006]: mgrmap e11: x(active, since 1.01636s) 2024-04-25T18:33:54.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-04-25T18:33:54.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T18:33:54.719 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.751+0000 7f08bb000700 1 Processor -- start 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.751+0000 7f08bb000700 1 -- start start 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.751+0000 7f08bb000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.752+0000 7f08bb000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08b407b7d0 con 0x7f08b407cda0 2024-04-25T18:33:54.720 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.752+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.752+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36386/0 (socket says 172.21.15.63:36386) 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.752+0000 7f08b3e00700 1 -- 172.21.15.63:0/1161312686 learned_addr learned my addr 172.21.15.63:0/1161312686 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.752+0000 7f08b3e00700 1 -- 172.21.15.63:0/1161312686 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b407b950 con 0x7f08b407cda0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.753+0000 7f08b3e00700 1 --2- 172.21.15.63:0/1161312686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08a40098a0 tx=0x7f08a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b37370de497 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.753+0000 7f08b2a00700 1 -- 172.21.15.63:0/1161312686 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4004160 con 0x7f08b407cda0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.753+0000 7f08b2a00700 1 -- 172.21.15.63:0/1161312686 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a4004300 con 0x7f08b407cda0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.753+0000 7f08b2a00700 1 -- 172.21.15.63:0/1161312686 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4011a20 con 0x7f08b407cda0 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 -- 172.21.15.63:0/1161312686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 msgr2=0x7f08b407b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 --2- 172.21.15.63:0/1161312686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08a40098a0 tx=0x7f08a4009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.722 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 -- 172.21.15.63:0/1161312686 shutdown_connections 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 --2- 172.21.15.63:0/1161312686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b407b200 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 -- 172.21.15.63:0/1161312686 >> 172.21.15.63:0/1161312686 conn(0x7f08b4103b50 msgr2=0x7f08b4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 -- 172.21.15.63:0/1161312686 shutdown_connections 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.754+0000 7f08bb000700 1 -- 172.21.15.63:0/1161312686 wait complete. 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.755+0000 7f08bb000700 1 Processor -- start 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.755+0000 7f08bb000700 1 -- start start 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.755+0000 7f08bb000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.755+0000 7f08bb000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08b410eb30 con 0x7f08b407cda0 2024-04-25T18:33:54.726 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.756+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.756+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36394/0 (socket says 172.21.15.63:36394) 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.756+0000 7f08b3e00700 1 -- 172.21.15.63:0/4166396635 learned_addr learned my addr 172.21.15.63:0/4166396635 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.756+0000 7f08b3e00700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b419eb40 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.756+0000 7f08b3e00700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009520 tx=0x7f08a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.757+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a40039e0 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.757+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a401b8f0 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.757+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b419edd0 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.757+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4019770 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.757+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b41a1ac0 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.758+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f08a4019a70 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.758+0000 7f08b0c00700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.758+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f089803c1d0 con 0x7f0898039600 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.758+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f08a404efe0 con 0x7f08b407cda0 2024-04-25T18:33:54.727 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.759+0000 7f08b3400700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.759+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.960+0000 7f08b3400700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:47.960+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:48.361+0000 7f08b3400700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:48.361+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:49.162+0000 7f08b3400700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:49.162+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:50.764+0000 7f08b3400700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3403991527 2024-04-25T18:33:54.728 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:50.764+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:52.966+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f08a401a8f0 con 0x7f08b407cda0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:52.966+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 msgr2=0x7f089803bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:52.966+0000 7f08b0c00700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.969+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f08a401a3d0 con 0x7f08b407cda0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.969+0000 7f08b0c00700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 0x7f089803f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.969+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f08a4024e70 con 0x7f089803cfe0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.971+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 0x7f089803f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.973+0000 7f08b3400700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 0x7f089803f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f08a0003a10 tx=0x7f08a00092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.974+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f08a4024e70 con 0x7f089803cfe0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.984+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f08b4069ba0 con 0x7f089803cfe0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.985+0000 7f08b0c00700 1 -- 172.21.15.63:0/4166396635 <== mgr.14152 v2:172.21.15.63:6800/3659595029 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f08b4069ba0 con 0x7f089803cfe0 2024-04-25T18:33:54.731 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.985+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 msgr2=0x7f089803f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.985+0000 7f08bb000700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 0x7f089803f3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f08a0003a10 tx=0x7f08a00092b0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.985+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 msgr2=0x7f08b419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.985+0000 7f08bb000700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009520 tx=0x7f08a4011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 shutdown_connections 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f089803cfe0 0x7f089803f3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b407cda0 0x7f08b419e600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 --2- 172.21.15.63:0/4166396635 >> [v2:172.21.15.63:6800/3403991527,v1:172.21.15.63:6801/3403991527] conn(0x7f0898039600 0x7f089803bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 >> 172.21.15.63:0/4166396635 conn(0x7f08b4103b50 msgr2=0x7f08b41045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 shutdown_connections 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:53.986+0000 7f08bb000700 1 -- 172.21.15.63:0/4166396635 wait complete. 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 9 is available 2024-04-25T18:33:54.732 INFO:teuthology.orchestra.run.smithi063.stdout:Generating a dashboard self-signed certificate... 2024-04-25T18:33:55.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:55 smithi063 ceph-mon[28006]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T18:33:55.153 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:55 smithi063 ceph-mon[28006]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T18:33:56.314 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:56 smithi063 ceph-mon[28006]: mgrmap e12: x(active, since 2s) 2024-04-25T18:33:57.254 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T18:33:57.254 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.052+0000 7ffb5fa00700 1 Processor -- start 2024-04-25T18:33:57.254 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.052+0000 7ffb5fa00700 1 -- start start 2024-04-25T18:33:57.254 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.052+0000 7ffb5fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.053+0000 7ffb5fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffb58063c80 con 0x7ffb58109090 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.053+0000 7ffb5d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.053+0000 7ffb5d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36458/0 (socket says 172.21.15.63:36458) 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.053+0000 7ffb5d000700 1 -- 172.21.15.63:0/1694976179 learned_addr learned my addr 172.21.15.63:0/1694976179 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.053+0000 7ffb5d000700 1 -- 172.21.15.63:0/1694976179 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5810b9c0 con 0x7ffb58109090 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.054+0000 7ffb5d000700 1 --2- 172.21.15.63:0/1694976179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0098a0 tx=0x7ffb4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beed5ba32683f8cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.054+0000 7ffb57400700 1 -- 172.21.15.63:0/1694976179 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c004160 con 0x7ffb58109090 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.054+0000 7ffb57400700 1 -- 172.21.15.63:0/1694976179 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb4c004300 con 0x7ffb58109090 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.054+0000 7ffb57400700 1 -- 172.21.15.63:0/1694976179 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c011a20 con 0x7ffb58109090 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 -- 172.21.15.63:0/1694976179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 msgr2=0x7ffb5810b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:57.260 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/1694976179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0098a0 tx=0x7ffb4c009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 -- 172.21.15.63:0/1694976179 shutdown_connections 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/1694976179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5810b480 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 -- 172.21.15.63:0/1694976179 >> 172.21.15.63:0/1694976179 conn(0x7ffb58102a40 msgr2=0x7ffb58104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 -- 172.21.15.63:0/1694976179 shutdown_connections 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.055+0000 7ffb5fa00700 1 -- 172.21.15.63:0/1694976179 wait complete. 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.056+0000 7ffb5fa00700 1 Processor -- start 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5fa00700 1 -- start start 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5810c900 con 0x7ffb58109090 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36462/0 (socket says 172.21.15.63:36462) 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.057+0000 7ffb5d000700 1 -- 172.21.15.63:0/2866870643 learned_addr learned my addr 172.21.15.63:0/2866870643 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:57.261 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.058+0000 7ffb5d000700 1 -- 172.21.15.63:0/2866870643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5819c940 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.058+0000 7ffb5d000700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c000c00 tx=0x7ffb4c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.059+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c003aa0 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.059+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb4c01b8f0 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.059+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb4c023870 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.059+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb5819cbd0 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.059+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb5819f8c0 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.060+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ffb4c01b440 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.060+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb580640a0 con 0x7ffb58109090 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.060+0000 7ffb55600700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 0x7ffb4003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:57.262 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.061+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ffb4c04dfa0 con 0x7ffb58109090 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.061+0000 7ffb57e00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 0x7ffb4003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.062+0000 7ffb57e00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 0x7ffb4003bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb44006fd0 tx=0x7ffb44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.068+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb4c01f030 con 0x7ffb58109090 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.312+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ffb58103ba0 con 0x7ffb40039660 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.432+0000 7ffb55600700 1 -- 172.21.15.63:0/2866870643 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ffb58103ba0 con 0x7ffb40039660 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 msgr2=0x7ffb4003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 0x7ffb4003bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb44006fd0 tx=0x7ffb44006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 msgr2=0x7ffb5819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c000c00 tx=0x7ffb4c011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 shutdown_connections 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffb40039660 0x7ffb4003bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 --2- 172.21.15.63:0/2866870643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb58109090 0x7ffb5819c400 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.435+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 >> 172.21.15.63:0/2866870643 conn(0x7ffb58102a40 msgr2=0x7ffb58103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.436+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 shutdown_connections 2024-04-25T18:33:57.263 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:56.436+0000 7ffb5fa00700 1 -- 172.21.15.63:0/2866870643 wait complete. 2024-04-25T18:33:57.264 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial admin user... 2024-04-25T18:33:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:33:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:57 smithi063 ceph-mon[28006]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:33:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:33:59.420 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$H0UsguzOGauEmp4jPUUNXOrwf2qlSNHZk7KxXxUVRmfbIyiDRpLnK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714070038, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.928+0000 7f08bac00700 1 Processor -- start 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.929+0000 7f08bac00700 1 -- start start 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.929+0000 7f08bac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.929+0000 7f08bac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08b4108f20 con 0x7f08b4108530 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.930+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.930+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46526/0 (socket says 172.21.15.63:46526) 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.930+0000 7f08b3e00700 1 -- 172.21.15.63:0/826363814 learned_addr learned my addr 172.21.15.63:0/826363814 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.930+0000 7f08b3e00700 1 -- 172.21.15.63:0/826363814 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b4109730 con 0x7f08b4108530 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.931+0000 7f08b3e00700 1 --2- 172.21.15.63:0/826363814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f08a40098a0 tx=0x7f08a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3650cd4b0c6973b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.931+0000 7f08b2a00700 1 -- 172.21.15.63:0/826363814 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4004160 con 0x7f08b4108530 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.931+0000 7f08b2a00700 1 -- 172.21.15.63:0/826363814 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a4004300 con 0x7f08b4108530 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.931+0000 7f08b2a00700 1 -- 172.21.15.63:0/826363814 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4011a20 con 0x7f08b4108530 2024-04-25T18:33:59.421 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 -- 172.21.15.63:0/826363814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 msgr2=0x7f08b4108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 --2- 172.21.15.63:0/826363814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f08a40098a0 tx=0x7f08a4009bb0 comp rx=0 tx=0).stop 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 -- 172.21.15.63:0/826363814 shutdown_connections 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 --2- 172.21.15.63:0/826363814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b4108950 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 -- 172.21.15.63:0/826363814 >> 172.21.15.63:0/826363814 conn(0x7f08b4103b50 msgr2=0x7f08b4105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 -- 172.21.15.63:0/826363814 shutdown_connections 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.932+0000 7f08bac00700 1 -- 172.21.15.63:0/826363814 wait complete. 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.933+0000 7f08bac00700 1 Processor -- start 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.933+0000 7f08bac00700 1 -- start start 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08bac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08bac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08b410cae0 con 0x7f08b4108530 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08b3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46540/0 (socket says 172.21.15.63:46540) 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08b3e00700 1 -- 172.21.15.63:0/2517239938 learned_addr learned my addr 172.21.15.63:0/2517239938 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:33:59.422 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.934+0000 7f08b3e00700 1 -- 172.21.15.63:0/2517239938 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b419eb90 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08b3e00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009520 tx=0x7f08a4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a40039e0 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a401b8f0 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b419ee20 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08a4019770 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.935+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b41a1b10 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.937+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f08a401b440 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.937+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b40640a0 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.937+0000 7f08b0c00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 0x7f089803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.937+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f08a404e080 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.937+0000 7f08b3400700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 0x7f089803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.938+0000 7f08b3400700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 0x7f089803bb10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08a0006fd0 tx=0x7f08a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:57.943+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08a401f030 con 0x7f08b4108530 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.198+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f08b419f300 con 0x7f0898039650 2024-04-25T18:33:59.423 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.487+0000 7f08b0c00700 1 -- 172.21.15.63:0/2517239938 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f08b419f300 con 0x7f0898039650 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 msgr2=0x7f089803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 0x7f089803bb10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f08a0006fd0 tx=0x7f08a0006e40 comp rx=0 tx=0).stop 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 msgr2=0x7f08b419e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f08a4009520 tx=0x7f08a4011aa0 comp rx=0 tx=0).stop 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 shutdown_connections 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0898039650 0x7f089803bb10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 --2- 172.21.15.63:0/2517239938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08b4108530 0x7f08b419e650 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 >> 172.21.15.63:0/2517239938 conn(0x7f08b4103b50 msgr2=0x7f08b4104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 shutdown_connections 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:33:58.490+0000 7f08bac00700 1 -- 172.21.15.63:0/2517239938 wait complete. 2024-04-25T18:33:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:Fetching dashboard port number... 2024-04-25T18:33:59.571 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:59 smithi063 ceph-mon[28006]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:33:59.571 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:33:59.571 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:33:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:00.550 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:00 smithi063 ceph-mon[28006]: mgrmap e13: x(active, since 6s) 2024-04-25T18:34:00.550 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:00 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/640441175' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.114+0000 7f7253200700 1 Processor -- start 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.114+0000 7f7253200700 1 -- start start 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.114+0000 7f7253200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.114+0000 7f7253200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f724c063d20 con 0x7f724c108ff0 2024-04-25T18:34:01.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.115+0000 7f724be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.115+0000 7f724be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46546/0 (socket says 172.21.15.63:46546) 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.115+0000 7f724be00700 1 -- 172.21.15.63:0/1543306020 learned_addr learned my addr 172.21.15.63:0/1543306020 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.115+0000 7f724be00700 1 -- 172.21.15.63:0/1543306020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724c10b920 con 0x7f724c108ff0 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.115+0000 7f724be00700 1 --2- 172.21.15.63:0/1543306020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f723c0098a0 tx=0x7f723c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259b4b6f56baca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.116+0000 7f724b400700 1 -- 172.21.15.63:0/1543306020 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f723c004160 con 0x7f724c108ff0 2024-04-25T18:34:01.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.116+0000 7f724b400700 1 -- 172.21.15.63:0/1543306020 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f723c004300 con 0x7f724c108ff0 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.116+0000 7f724b400700 1 -- 172.21.15.63:0/1543306020 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f723c011a20 con 0x7f724c108ff0 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 -- 172.21.15.63:0/1543306020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 msgr2=0x7f724c10b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 --2- 172.21.15.63:0/1543306020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f723c0098a0 tx=0x7f723c009bb0 comp rx=0 tx=0).stop 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 -- 172.21.15.63:0/1543306020 shutdown_connections 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 --2- 172.21.15.63:0/1543306020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c10b3e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:01.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 -- 172.21.15.63:0/1543306020 >> 172.21.15.63:0/1543306020 conn(0x7f724c102a40 msgr2=0x7f724c104e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 -- 172.21.15.63:0/1543306020 shutdown_connections 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.117+0000 7f7253200700 1 -- 172.21.15.63:0/1543306020 wait complete. 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.118+0000 7f7253200700 1 Processor -- start 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.118+0000 7f7253200700 1 -- start start 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f7253200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:01.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f7253200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f724c10c920 con 0x7f724c108ff0 2024-04-25T18:34:01.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f724be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:01.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f724be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46560/0 (socket says 172.21.15.63:46560) 2024-04-25T18:34:01.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f724be00700 1 -- 172.21.15.63:0/640441175 learned_addr learned my addr 172.21.15.63:0/640441175 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:01.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.119+0000 7f724be00700 1 -- 172.21.15.63:0/640441175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724c19c920 con 0x7f724c108ff0 2024-04-25T18:34:01.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f724be00700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f723c000c00 tx=0x7f723c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:01.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f723c003aa0 con 0x7f724c108ff0 2024-04-25T18:34:01.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f723c01b8f0 con 0x7f724c108ff0 2024-04-25T18:34:01.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f723c023870 con 0x7f724c108ff0 2024-04-25T18:34:01.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724c19cbb0 con 0x7f724c108ff0 2024-04-25T18:34:01.625 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.120+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724c19f8a0 con 0x7f724c108ff0 2024-04-25T18:34:01.625 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.122+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f724c0640a0 con 0x7f724c108ff0 2024-04-25T18:34:01.625 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.122+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f723c023b70 con 0x7f724c108ff0 2024-04-25T18:34:01.625 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.122+0000 7f7249600700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 0x7f723003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:01.626 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.122+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f723c04e170 con 0x7f724c108ff0 2024-04-25T18:34:01.626 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.123+0000 7f7243e00700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 0x7f723003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:01.626 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.124+0000 7f7243e00700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 0x7f723003bb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7234006fd0 tx=0x7f7234006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:01.626 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.129+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f723c01f030 con 0x7f724c108ff0 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.363+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f724c19fe90 con 0x7f724c108ff0 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.363+0000 7f7249600700 1 -- 172.21.15.63:0/640441175 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f723c0292a0 con 0x7f724c108ff0 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 msgr2=0x7f723003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 0x7f723003bb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7234006fd0 tx=0x7f7234006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 msgr2=0x7f724c19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:01.627 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f723c000c00 tx=0x7f723c011aa0 comp rx=0 tx=0).stop 2024-04-25T18:34:01.628 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 shutdown_connections 2024-04-25T18:34:01.628 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f72300396a0 0x7f723003bb60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:01.628 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 --2- 172.21.15.63:0/640441175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f724c108ff0 0x7f724c19c3e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:01.628 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 >> 172.21.15.63:0/640441175 conn(0x7f724c102a40 msgr2=0x7f724c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:01.629 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.367+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 shutdown_connections 2024-04-25T18:34:01.629 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:00.368+0000 7f7253200700 1 -- 172.21.15.63:0/640441175 wait complete. 2024-04-25T18:34:01.652 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T18:34:01.652 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-04-25T18:34:01.676 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T18:34:01.677 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-04-25T18:34:01.677 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-04-25T18:34:01.677 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T18:34:01.679 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph Dashboard is now available at: 2024-04-25T18:34:01.680 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:01.680 INFO:teuthology.orchestra.run.smithi063.stdout: URL: https://smithi063.front.sepia.ceph.com:8443/ 2024-04-25T18:34:01.680 INFO:teuthology.orchestra.run.smithi063.stdout: User: admin 2024-04-25T18:34:01.680 INFO:teuthology.orchestra.run.smithi063.stdout: Password: 6g8dn09qg4 2024-04-25T18:34:01.680 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:01.681 INFO:teuthology.orchestra.run.smithi063.stdout:Saving cluster configuration to /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config directory 2024-04-25T18:34:01.682 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling autotune for osd_memory_target 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.722+0000 7ff86fa00700 1 Processor -- start 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.722+0000 7ff86fa00700 1 -- start start 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.723+0000 7ff86fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.723+0000 7ff86fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff86807b790 con 0x7ff86807cd60 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.723+0000 7ff86d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.723+0000 7ff86d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46564/0 (socket says 172.21.15.63:46564) 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.723+0000 7ff86d000700 1 -- 172.21.15.63:0/828949204 learned_addr learned my addr 172.21.15.63:0/828949204 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.724+0000 7ff86d000700 1 -- 172.21.15.63:0/828949204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86807b910 con 0x7ff86807cd60 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.724+0000 7ff86d000700 1 --2- 172.21.15.63:0/828949204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098a0 tx=0x7ff858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1277b45a97e4c641 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.724+0000 7ff867e00700 1 -- 172.21.15.63:0/828949204 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff858004160 con 0x7ff86807cd60 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.724+0000 7ff867e00700 1 -- 172.21.15.63:0/828949204 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff858004300 con 0x7ff86807cd60 2024-04-25T18:34:03.695 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.724+0000 7ff867e00700 1 -- 172.21.15.63:0/828949204 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff858011a20 con 0x7ff86807cd60 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 -- 172.21.15.63:0/828949204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 msgr2=0x7ff86807b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 --2- 172.21.15.63:0/828949204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098a0 tx=0x7ff858009bb0 comp rx=0 tx=0).stop 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 -- 172.21.15.63:0/828949204 shutdown_connections 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 --2- 172.21.15.63:0/828949204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86807b1c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 -- 172.21.15.63:0/828949204 >> 172.21.15.63:0/828949204 conn(0x7ff868103b50 msgr2=0x7ff868105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 -- 172.21.15.63:0/828949204 shutdown_connections 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.725+0000 7ff86fa00700 1 -- 172.21.15.63:0/828949204 wait complete. 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.726+0000 7ff86fa00700 1 Processor -- start 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.726+0000 7ff86fa00700 1 -- start start 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff86810eb10 con 0x7ff86807cd60 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46568/0 (socket says 172.21.15.63:46568) 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86d000700 1 -- 172.21.15.63:0/1751666131 learned_addr learned my addr 172.21.15.63:0/1751666131 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:03.696 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.727+0000 7ff86d000700 1 -- 172.21.15.63:0/1751666131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86819eb10 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff86d000700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff858011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff858003aa0 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff85801b8f0 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff858023870 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86819eda0 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.728+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8681a1a90 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.729+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff85801b440 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.729+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8680640a0 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.730+0000 7ff866000700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 0x7ff84c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.730+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff85804e090 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.730+0000 7ff85fe00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 0x7ff84c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.731+0000 7ff85fe00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 0x7ff84c03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff850006fd0 tx=0x7ff850006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.736+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff85801f030 con 0x7ff86807cd60 2024-04-25T18:34:03.697 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.966+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7ff8681a20f0 con 0x7ff86807cd60 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.967+0000 7ff866000700 1 -- 172.21.15.63:0/1751666131 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7ff85800e260 con 0x7ff86807cd60 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 msgr2=0x7ff84c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 0x7ff84c03bbc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff850006fd0 tx=0x7ff850006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 msgr2=0x7ff86819e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff858011aa0 comp rx=0 tx=0).stop 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 shutdown_connections 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff84c039700 0x7ff84c03bbc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 --2- 172.21.15.63:0/1751666131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff86807cd60 0x7ff86819e5d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.969+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 >> 172.21.15.63:0/1751666131 conn(0x7ff868103b50 msgr2=0x7ff8681045a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.970+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 shutdown_connections 2024-04-25T18:34:03.698 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:02.970+0000 7ff86fa00700 1 -- 172.21.15.63:0/1751666131 wait complete. 2024-04-25T18:34:05.488 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.368+0000 7f0ceda00700 1 Processor -- start 2024-04-25T18:34:05.588 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.368+0000 7f0ceda00700 1 -- start start 2024-04-25T18:34:05.588 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.368+0000 7f0ceda00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:05.588 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.368+0000 7f0ceda00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce810a390 con 0x7f0ce810be00 2024-04-25T18:34:05.588 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.369+0000 7f0ce6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.369+0000 7f0ce6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46578/0 (socket says 172.21.15.63:46578) 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.369+0000 7f0ce6a00700 1 -- 172.21.15.63:0/2624766087 learned_addr learned my addr 172.21.15.63:0/2624766087 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.369+0000 7f0ce6a00700 1 -- 172.21.15.63:0/2624766087 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce810a510 con 0x7f0ce810be00 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.370+0000 7f0ce6a00700 1 --2- 172.21.15.63:0/2624766087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80098a0 tx=0x7f0cd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6d410af3e1454b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.370+0000 7f0ce5600700 1 -- 172.21.15.63:0/2624766087 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cd8004160 con 0x7f0ce810be00 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.370+0000 7f0ce5600700 1 -- 172.21.15.63:0/2624766087 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cd8004300 con 0x7f0ce810be00 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.370+0000 7f0ce5600700 1 -- 172.21.15.63:0/2624766087 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cd8011a20 con 0x7f0ce810be00 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 -- 172.21.15.63:0/2624766087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 msgr2=0x7f0ce810c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:05.589 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 --2- 172.21.15.63:0/2624766087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80098a0 tx=0x7f0cd8009bb0 comp rx=0 tx=0).stop 2024-04-25T18:34:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 -- 172.21.15.63:0/2624766087 shutdown_connections 2024-04-25T18:34:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 --2- 172.21.15.63:0/2624766087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce810c220 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:05.590 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 -- 172.21.15.63:0/2624766087 >> 172.21.15.63:0/2624766087 conn(0x7f0ce807dc00 msgr2=0x7f0ce81084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:05.595 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 -- 172.21.15.63:0/2624766087 shutdown_connections 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.371+0000 7f0ceda00700 1 -- 172.21.15.63:0/2624766087 wait complete. 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.372+0000 7f0ceda00700 1 Processor -- start 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.372+0000 7f0ceda00700 1 -- start start 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ceda00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ceda00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce810ecd0 con 0x7f0ce810be00 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ce6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ce6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46582/0 (socket says 172.21.15.63:46582) 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ce6a00700 1 -- 172.21.15.63:0/1196339258 learned_addr learned my addr 172.21.15.63:0/1196339258 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.373+0000 7f0ce6a00700 1 -- 172.21.15.63:0/1196339258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce819ec90 con 0x7f0ce810be00 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0ce6a00700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8000c00 tx=0x7f0cd8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:05.596 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cd8003aa0 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cd801b8f0 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce819ef20 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0cd8023890 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.374+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce81a1c10 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.375+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ce810ee50 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.375+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0cd801b440 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.376+0000 7f0cdf400700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 0x7f0ccc03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.376+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0cd804e070 con 0x7f0ce810be00 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.376+0000 7f0ce6000700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 0x7f0ccc03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:05.597 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.377+0000 7f0ce6000700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 0x7f0ccc03bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4006fd0 tx=0x7f0cd4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.382+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0cd801f030 con 0x7f0ce810be00 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.738+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f0ce81a1de0 con 0x7f0ce810be00 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.743+0000 7f0cdf400700 1 -- 172.21.15.63:0/1196339258 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f0cd8016260 con 0x7f0ce810be00 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 msgr2=0x7f0ccc03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 0x7f0ccc03bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4006fd0 tx=0x7f0cd4006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 msgr2=0x7f0ce819e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8000c00 tx=0x7f0cd8011aa0 comp rx=0 tx=0).stop 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 shutdown_connections 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0ccc039700 0x7f0ccc03bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 --2- 172.21.15.63:0/1196339258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ce810be00 0x7f0ce819e750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 >> 172.21.15.63:0/1196339258 conn(0x7f0ce807dc00 msgr2=0x7f0ce8107bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 shutdown_connections 2024-04-25T18:34:05.598 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-04-25T18:34:04.746+0000 7f0ceda00700 1 -- 172.21.15.63:0/1196339258 wait complete. 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: ceph telemetry on 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi063.stdout:For more information see: 2024-04-25T18:34:05.600 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.600 INFO:teuthology.orchestra.run.smithi063.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T18:34:05.600 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:05.600 INFO:teuthology.orchestra.run.smithi063.stdout:Bootstrap complete. 2024-04-25T18:34:05.603 INFO:tasks.cephadm:Fetching config... 2024-04-25T18:34:05.604 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:34:05.604 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T18:34:05.695 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T18:34:05.695 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:34:05.695 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T18:34:05.737 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T18:34:05.738 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:34:05.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T18:34:05.861 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T18:34:05.861 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:34:05.861 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T18:34:05.912 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T18:34:05.913 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T18:34:05.995 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:05 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1196339258' entity='client.admin' 2024-04-25T18:34:06.038 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:34:06.074 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T18:34:06.116 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:34:06.127 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T18:34:06.169 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvr5SgsR96G3VTAxHfSKMh69zb4Vq+JTYMQAfvis5OSXd7d0misuRERDgF14aBOy6t0/Qg46exviisJYYpNKFjCK6486YdXKGTYYLTN+Wc4ab65moANQ8Zxd3k0/c36RMffCzcLsZuliZ4x52Fkcsfyvoxpf9XTxHNb2OHjRllG+UhVguspKJq/MSrkI8mUDCDpUE5ctkKE3N0WPseSbN6Us6Wrfjwb8QQS4wReHP+U9urfve++g3vN0AnooR0UmRNmT+pJ6F0EIwjlfiKY5PNnzkbo1xVZIK4vN4J1cRj0Hvnbgzway5IW0V/f7z7vPDQzQmNXKJENo+ksldFXCgXwwwKVPXfq47AXZtMpGjlBDs58AzakKh43Y9opDoMO3UmATRtgUGiF31h6YhVm9dTYcxWbDL3gC5UuZLlPKlHWmI8MLTz+Q5lSLCx0iwrSCnmpCKBMo4b/DMrhSAmkbpvizaPfmFodR9H96MGgXo6Ppapj8noTnCmlX+ck5KqQKc= ceph-e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:34:06.181 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T18:34:06.606 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 -- 172.21.15.63:0/1645662203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 msgr2=0x7ff6dc103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/1645662203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc103fd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009910 tx=0x7ff6d0009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 -- 172.21.15.63:0/1645662203 shutdown_connections 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/1645662203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc103fd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 -- 172.21.15.63:0/1645662203 >> 172.21.15.63:0/1645662203 conn(0x7ff6dc0fda90 msgr2=0x7ff6dc0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 -- 172.21.15.63:0/1645662203 shutdown_connections 2024-04-25T18:34:09.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.041+0000 7ff6e3c00700 1 -- 172.21.15.63:0/1645662203 wait complete. 2024-04-25T18:34:09.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.042+0000 7ff6e3c00700 1 Processor -- start 2024-04-25T18:34:09.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e3c00700 1 -- start start 2024-04-25T18:34:09.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e3c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:09.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e3c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc108520 con 0x7ff6dc103bb0 2024-04-25T18:34:09.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:09.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43026/0 (socket says 172.21.15.63:43026) 2024-04-25T18:34:09.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.043+0000 7ff6e1200700 1 -- 172.21.15.63:0/597563196 learned_addr learned my addr 172.21.15.63:0/597563196 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:09.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.044+0000 7ff6e1200700 1 -- 172.21.15.63:0/597563196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc19a350 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.044+0000 7ff6e1200700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0005950 tx=0x7ff6d0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.044+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d0026070 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.044+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d000bc00 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.044+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc19a5e0 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.045+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6d0003a10 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.045+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc19d2d0 con 0x7ff6dc103bb0 2024-04-25T18:34:09.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.046+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff6d0029070 con 0x7ff6dc103bb0 2024-04-25T18:34:09.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.046+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc108ca0 con 0x7ff6dc103bb0 2024-04-25T18:34:09.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.046+0000 7ff6d9600700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 0x7ff6c403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:09.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.046+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff6d001e070 con 0x7ff6dc103bb0 2024-04-25T18:34:09.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.047+0000 7ff6dbe00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 0x7ff6c403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:09.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.047+0000 7ff6dbe00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 0x7ff6c403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:09.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.053+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6d0013070 con 0x7ff6dc103bb0 2024-04-25T18:34:09.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.289+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7ff6dc02df80 con 0x7ff6dc103bb0 2024-04-25T18:34:09.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.295+0000 7ff6d9600700 1 -- 172.21.15.63:0/597563196 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7ff6d00300c0 con 0x7ff6dc103bb0 2024-04-25T18:34:09.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 msgr2=0x7ff6c403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 0x7ff6c403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 msgr2=0x7ff6dc199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0005950 tx=0x7ff6d0004fa0 comp rx=0 tx=0).stop 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 shutdown_connections 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff6c40396b0 0x7ff6c403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 --2- 172.21.15.63:0/597563196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6dc103bb0 0x7ff6dc199e10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:09.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.299+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 >> 172.21.15.63:0/597563196 conn(0x7ff6dc0fda90 msgr2=0x7ff6dc0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:09.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.300+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 shutdown_connections 2024-04-25T18:34:09.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:09.300+0000 7ff6e3c00700 1 -- 172.21.15.63:0/597563196 wait complete. 2024-04-25T18:34:10.147 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T18:34:10.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T18:34:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/597563196' entity='client.admin' 2024-04-25T18:34:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:34:10.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:10.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:10.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:13.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 -- 172.21.15.63:0/3420631180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 msgr2=0x7f2930104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 --2- 172.21.15.63:0/3420631180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930104ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2920009910 tx=0x7f2920009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 -- 172.21.15.63:0/3420631180 shutdown_connections 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 --2- 172.21.15.63:0/3420631180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930104ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 -- 172.21.15.63:0/3420631180 >> 172.21.15.63:0/3420631180 conn(0x7f29300ffc60 msgr2=0x7f29301020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 -- 172.21.15.63:0/3420631180 shutdown_connections 2024-04-25T18:34:13.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.706+0000 7f2937000700 1 -- 172.21.15.63:0/3420631180 wait complete. 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.707+0000 7f2937000700 1 Processor -- start 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.707+0000 7f2937000700 1 -- start start 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.707+0000 7f2937000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.707+0000 7f2937000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f293010a600 con 0x7f29301046c0 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.708+0000 7f292fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.708+0000 7f292fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43044/0 (socket says 172.21.15.63:43044) 2024-04-25T18:34:13.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.708+0000 7f292fe00700 1 -- 172.21.15.63:0/779678423 learned_addr learned my addr 172.21.15.63:0/779678423 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.708+0000 7f292fe00700 1 -- 172.21.15.63:0/779678423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293019a250 con 0x7f29301046c0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.708+0000 7f292fe00700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2920005800 tx=0x7f2920009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.709+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2920027070 con 0x7f29301046c0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.709+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f292000be10 con 0x7f29301046c0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.709+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f293019a4e0 con 0x7f29301046c0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.709+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f292000f460 con 0x7f29301046c0 2024-04-25T18:34:13.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.709+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293019d180 con 0x7f29301046c0 2024-04-25T18:34:13.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.710+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f292002a070 con 0x7f29301046c0 2024-04-25T18:34:13.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.710+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f293010ad80 con 0x7f29301046c0 2024-04-25T18:34:13.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.711+0000 7f292cc00700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 0x7f291403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:13.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.711+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f292001e030 con 0x7f29301046c0 2024-04-25T18:34:13.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.711+0000 7f292f400700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 0x7f291403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:13.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.712+0000 7f292f400700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 0x7f291403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f291c006fd0 tx=0x7f291c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:13.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.717+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2920013070 con 0x7f29301046c0 2024-04-25T18:34:13.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.952+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f293002e120 con 0x7f2914039650 2024-04-25T18:34:13.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.958+0000 7f292cc00700 1 -- 172.21.15.63:0/779678423 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f293002e120 con 0x7f2914039650 2024-04-25T18:34:13.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 msgr2=0x7f291403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 0x7f291403bb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f291c006fd0 tx=0x7f291c006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 msgr2=0x7f2930199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2920005800 tx=0x7f2920009f60 comp rx=0 tx=0).stop 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 shutdown_connections 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2914039650 0x7f291403bb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 --2- 172.21.15.63:0/779678423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29301046c0 0x7f2930199d10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.961+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 >> 172.21.15.63:0/779678423 conn(0x7f29300ffc60 msgr2=0x7f2930100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.962+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 shutdown_connections 2024-04-25T18:34:13.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:13.962+0000 7f2937000700 1 -- 172.21.15.63:0/779678423 wait complete. 2024-04-25T18:34:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:14 smithi063 ceph-mon[28006]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:34:14.828 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi070 2024-04-25T18:34:14.828 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:34:14.828 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.conf 2024-04-25T18:34:14.847 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:34:14.847 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:14.904 INFO:tasks.cephadm:Adding host smithi070 to orchestrator... 2024-04-25T18:34:14.904 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch host add smithi070 2024-04-25T18:34:15.328 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:15.376 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:15 smithi063 ceph-mon[28006]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:34:15.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:15 smithi063 ceph-mon[28006]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:34:15.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:15 smithi063 ceph-mon[28006]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:34:16.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:16 smithi063 ceph-mon[28006]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:16.370 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:16 smithi063 ceph-mon[28006]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:17.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 -- 172.21.15.63:0/2258056596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 msgr2=0x7f8b4010dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 --2- 172.21.15.63:0/2258056596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4010dd50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b34009910 tx=0x7f8b34009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 -- 172.21.15.63:0/2258056596 shutdown_connections 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 --2- 172.21.15.63:0/2258056596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4010dd50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 -- 172.21.15.63:0/2258056596 >> 172.21.15.63:0/2258056596 conn(0x7f8b400782d0 msgr2=0x7f8b4007a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 -- 172.21.15.63:0/2258056596 shutdown_connections 2024-04-25T18:34:17.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.539+0000 7f8b48200700 1 -- 172.21.15.63:0/2258056596 wait complete. 2024-04-25T18:34:17.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.540+0000 7f8b48200700 1 Processor -- start 2024-04-25T18:34:17.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b48200700 1 -- start start 2024-04-25T18:34:17.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b48200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:17.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b48200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b40112f70 con 0x7f8b4010d970 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b45800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b45800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34652/0 (socket says 172.21.15.63:34652) 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.541+0000 7f8b45800700 1 -- 172.21.15.63:0/3196280330 learned_addr learned my addr 172.21.15.63:0/3196280330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b45800700 1 -- 172.21.15.63:0/3196280330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4019e9c0 con 0x7f8b4010d970 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b45800700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b34006010 tx=0x7f8b34005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:17.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b34026070 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b3400bba0 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b34003930 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.542+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4019ecb0 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.543+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b401a2d50 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.544+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8b34029070 con 0x7f8b4010d970 2024-04-25T18:34:17.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.544+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4010a190 con 0x7f8b4010d970 2024-04-25T18:34:17.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.544+0000 7f8b3e000700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 0x7f8b2803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:17.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.544+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b3401e070 con 0x7f8b4010d970 2024-04-25T18:34:17.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.545+0000 7f8b44e00700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 0x7f8b2803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:17.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.545+0000 7f8b44e00700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 0x7f8b2803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c006fd0 tx=0x7f8b2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:17.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.550+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b34013070 con 0x7f8b4010d970 2024-04-25T18:34:17.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:17.785+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}) -- 0x7f8b4019f7e0 con 0x7f8b28039650 2024-04-25T18:34:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:17 smithi063 ceph-mon[28006]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:34:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:18.842 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:18 smithi063 ceph-mon[28006]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:18.842 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:18 smithi063 ceph-mon[28006]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:34:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:19 smithi063 ceph-mon[28006]: Deploying cephadm binary to smithi070 2024-04-25T18:34:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:20 smithi063 ceph-mon[28006]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:22.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.637+0000 7f8b3e000700 1 -- 172.21.15.63:0/3196280330 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8b4019f7e0 con 0x7f8b28039650 2024-04-25T18:34:22.640 INFO:teuthology.orchestra.run.smithi063.stdout:Added host 'smithi070' with addr '172.21.15.70' 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.640+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 msgr2=0x7f8b2803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.640+0000 7f8b48200700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 0x7f8b2803bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c006fd0 tx=0x7f8b2c006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 msgr2=0x7f8b4019e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b34006010 tx=0x7f8b34005780 comp rx=0 tx=0).stop 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 shutdown_connections 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8b28039650 0x7f8b2803bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 --2- 172.21.15.63:0/3196280330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b4010d970 0x7f8b4019e480 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:22.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 >> 172.21.15.63:0/3196280330 conn(0x7f8b400782d0 msgr2=0x7f8b40110560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:22.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 shutdown_connections 2024-04-25T18:34:22.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:22.641+0000 7f8b48200700 1 -- 172.21.15.63:0/3196280330 wait complete. 2024-04-25T18:34:23.033 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:22 smithi063 ceph-mon[28006]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:23.034 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:23.034 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:23.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T18:34:23.899 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:24.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:23 smithi063 ceph-mon[28006]: Added host smithi070 2024-04-25T18:34:24.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:25.117 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:24 smithi063 ceph-mon[28006]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:25.117 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:24 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:26.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 -- 172.21.15.63:0/953795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 msgr2=0x7f526410dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 --2- 172.21.15.63:0/953795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526410dd40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5258009910 tx=0x7f5258009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 -- 172.21.15.63:0/953795679 shutdown_connections 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 --2- 172.21.15.63:0/953795679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526410dd40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 -- 172.21.15.63:0/953795679 >> 172.21.15.63:0/953795679 conn(0x7f52640781e0 msgr2=0x7f526407a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 -- 172.21.15.63:0/953795679 shutdown_connections 2024-04-25T18:34:26.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.055+0000 7f526c200700 1 -- 172.21.15.63:0/953795679 wait complete. 2024-04-25T18:34:26.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.057+0000 7f526c200700 1 Processor -- start 2024-04-25T18:34:26.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.057+0000 7f526c200700 1 -- start start 2024-04-25T18:34:26.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.057+0000 7f526c200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:26.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.057+0000 7f526c200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5264112f60 con 0x7f526410d960 2024-04-25T18:34:26.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.058+0000 7f5269800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:26.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.058+0000 7f5269800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34680/0 (socket says 172.21.15.63:34680) 2024-04-25T18:34:26.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.058+0000 7f5269800700 1 -- 172.21.15.63:0/3828903798 learned_addr learned my addr 172.21.15.63:0/3828903798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:26.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.058+0000 7f5269800700 1 -- 172.21.15.63:0/3828903798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526419e9b0 con 0x7f526410d960 2024-04-25T18:34:26.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f5269800700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52580056d0 tx=0x7f52580057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:26.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5258026070 con 0x7f526410d960 2024-04-25T18:34:26.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5258004370 con 0x7f526410d960 2024-04-25T18:34:26.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52580039b0 con 0x7f526410d960 2024-04-25T18:34:26.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526419eca0 con 0x7f526410d960 2024-04-25T18:34:26.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.059+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52641a2d40 con 0x7f526410d960 2024-04-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.060+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5258029070 con 0x7f526410d960 2024-04-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.060+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f526410a180 con 0x7f526410d960 2024-04-25T18:34:26.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.061+0000 7f5262000700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 0x7f524c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.061+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f525801e070 con 0x7f526410d960 2024-04-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.061+0000 7f5268e00700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 0x7f524c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:26.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.062+0000 7f5268e00700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 0x7f524c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:26.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.067+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5258013070 con 0x7f526410d960 2024-04-25T18:34:26.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.303+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f526419f7d0 con 0x7f524c0396b0 2024-04-25T18:34:26.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.306+0000 7f5262000700 1 -- 172.21.15.63:0/3828903798 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f526419f7d0 con 0x7f524c0396b0 2024-04-25T18:34:26.307 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:26.307 INFO:teuthology.orchestra.run.smithi063.stdout:[{"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}] 2024-04-25T18:34:26.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 msgr2=0x7f524c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:26.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 0x7f524c03bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5250006fd0 tx=0x7f5250006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 msgr2=0x7f526419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52580056d0 tx=0x7f52580057b0 comp rx=0 tx=0).stop 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 shutdown_connections 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f524c0396b0 0x7f524c03bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 --2- 172.21.15.63:0/3828903798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f526410d960 0x7f526419e470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.309+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 >> 172.21.15.63:0/3828903798 conn(0x7f52640781e0 msgr2=0x7f5264078f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.310+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 shutdown_connections 2024-04-25T18:34:26.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:26.310+0000 7f526c200700 1 -- 172.21.15.63:0/3828903798 wait complete. 2024-04-25T18:34:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:26 smithi063 ceph-mon[28006]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:27.054 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2024-04-25T18:34:27.054 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:34:27.054 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2024-04-25T18:34:27.072 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:34:27.072 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:27.129 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2024-04-25T18:34:27.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch host add smithi114 2024-04-25T18:34:27.578 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:27.626 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:27 smithi063 ceph-mon[28006]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:34:28.581 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:28 smithi063 ceph-mon[28006]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 -- 172.21.15.63:0/996079497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 msgr2=0x7efec006a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 --2- 172.21.15.63:0/996079497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec006a240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009910 tx=0x7efeb0009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 -- 172.21.15.63:0/996079497 shutdown_connections 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 --2- 172.21.15.63:0/996079497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec006a240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 -- 172.21.15.63:0/996079497 >> 172.21.15.63:0/996079497 conn(0x7efec0076ea0 msgr2=0x7efec00772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 -- 172.21.15.63:0/996079497 shutdown_connections 2024-04-25T18:34:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.810+0000 7efec6c00700 1 -- 172.21.15.63:0/996079497 wait complete. 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.811+0000 7efec6c00700 1 Processor -- start 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efec6c00700 1 -- start start 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efec6c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efec6c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efec010eb90 con 0x7efec0069e60 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efebfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efebfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57158/0 (socket says 172.21.15.63:57158) 2024-04-25T18:34:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.812+0000 7efebfe00700 1 -- 172.21.15.63:0/3760976141 learned_addr learned my addr 172.21.15.63:0/3760976141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.813+0000 7efebfe00700 1 -- 172.21.15.63:0/3760976141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec019a640 con 0x7efec0069e60 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.813+0000 7efebfe00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006010 tx=0x7efeb0005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.813+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb0026070 con 0x7efec0069e60 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.814+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efeb000bc00 con 0x7efec0069e60 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.814+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec019a8d0 con 0x7efec0069e60 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.814+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efeb0003910 con 0x7efec0069e60 2024-04-25T18:34:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.814+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec019ad30 con 0x7efec0069e60 2024-04-25T18:34:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.815+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efeb0029070 con 0x7efec0069e60 2024-04-25T18:34:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.815+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec0101970 con 0x7efec0069e60 2024-04-25T18:34:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.815+0000 7efebcc00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 0x7efea403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.815+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efeb001e070 con 0x7efec0069e60 2024-04-25T18:34:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.816+0000 7efebf400700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 0x7efea403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.816+0000 7efebf400700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 0x7efea403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:29.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:29.822+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efeb0013070 con 0x7efec0069e60 2024-04-25T18:34:30.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:30.067+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}) -- 0x7efec019b4d0 con 0x7efea40396b0 2024-04-25T18:34:30.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:30.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:30.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:30.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:30.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:34:30.069 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:30.069 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:34:30.069 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:29 smithi063 ceph-mon[28006]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:34:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:34:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:34:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:32.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:31 smithi063 ceph-mon[28006]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:34:32.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:31 smithi063 ceph-mon[28006]: Deploying cephadm binary to smithi114 2024-04-25T18:34:33.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:32 smithi063 ceph-mon[28006]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:34.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.924+0000 7efebcc00700 1 -- 172.21.15.63:0/3760976141 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7efec019b4d0 con 0x7efea40396b0 2024-04-25T18:34:34.925 INFO:teuthology.orchestra.run.smithi063.stdout:Added host 'smithi114' with addr '172.21.15.114' 2024-04-25T18:34:34.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 msgr2=0x7efea403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:34.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 0x7efea403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:34.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 msgr2=0x7efec019a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:34.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efeb0006010 tx=0x7efeb0005780 comp rx=0 tx=0).stop 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 shutdown_connections 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efea40396b0 0x7efea403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 --2- 172.21.15.63:0/3760976141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec0069e60 0x7efec019a100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 >> 172.21.15.63:0/3760976141 conn(0x7efec0076ea0 msgr2=0x7efec01099c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.928+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 shutdown_connections 2024-04-25T18:34:34.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:34.929+0000 7efec6c00700 1 -- 172.21.15.63:0/3760976141 wait complete. 2024-04-25T18:34:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:34 smithi063 ceph-mon[28006]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:35.713 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T18:34:36.164 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:36.311 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:35 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:36.311 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:35 smithi063 ceph-mon[28006]: Added host smithi114 2024-04-25T18:34:36.311 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:35 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:36.311 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:35 smithi063 ceph-mon[28006]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:36.311 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:35 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:38.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.432+0000 7fb9a2800700 1 -- 172.21.15.63:0/3044489752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 msgr2=0x7fb99c1088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:38.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.432+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3044489752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c1088c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009910 tx=0x7fb98c009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:38.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.432+0000 7fb9a2800700 1 -- 172.21.15.63:0/3044489752 shutdown_connections 2024-04-25T18:34:38.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.432+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3044489752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c1088c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:38.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.432+0000 7fb9a2800700 1 -- 172.21.15.63:0/3044489752 >> 172.21.15.63:0/3044489752 conn(0x7fb99c0fdf80 msgr2=0x7fb99c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:38.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.433+0000 7fb9a2800700 1 -- 172.21.15.63:0/3044489752 shutdown_connections 2024-04-25T18:34:38.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.433+0000 7fb9a2800700 1 -- 172.21.15.63:0/3044489752 wait complete. 2024-04-25T18:34:38.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.434+0000 7fb9a2800700 1 Processor -- start 2024-04-25T18:34:38.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.434+0000 7fb9a2800700 1 -- start start 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.435+0000 7fb9a2800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.435+0000 7fb9a2800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c10c940 con 0x7fb99c1084e0 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.435+0000 7fb99b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.435+0000 7fb99b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45230/0 (socket says 172.21.15.63:45230) 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.435+0000 7fb99b400700 1 -- 172.21.15.63:0/3104842323 learned_addr learned my addr 172.21.15.63:0/3104842323 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:38.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb99b400700 1 -- 172.21.15.63:0/3104842323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c19a620 con 0x7fb99c1084e0 2024-04-25T18:34:38.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb99b400700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006010 tx=0x7fb98c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:38.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c026070 con 0x7fb99c1084e0 2024-04-25T18:34:38.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c00bba0 con 0x7fb99c1084e0 2024-04-25T18:34:38.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb98c003910 con 0x7fb99c1084e0 2024-04-25T18:34:38.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.436+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c19a910 con 0x7fb99c1084e0 2024-04-25T18:34:38.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.437+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c19e9b0 con 0x7fb99c1084e0 2024-04-25T18:34:38.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.438+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c103b90 con 0x7fb99c1084e0 2024-04-25T18:34:38.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.438+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb98c029070 con 0x7fb99c1084e0 2024-04-25T18:34:38.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.438+0000 7fb993e00700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 0x7fb98003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:38.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.439+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb98c01e070 con 0x7fb99c1084e0 2024-04-25T18:34:38.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.439+0000 7fb99aa00700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 0x7fb98003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:38.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.440+0000 7fb99aa00700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 0x7fb98003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb988006fd0 tx=0x7fb988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:38.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.445+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb98c013070 con 0x7fb99c1084e0 2024-04-25T18:34:38.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.682+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb99c19b440 con 0x7fb980039650 2024-04-25T18:34:38.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.685+0000 7fb993e00700 1 -- 172.21.15.63:0/3104842323 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fb99c19b440 con 0x7fb980039650 2024-04-25T18:34:38.687 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:34:38.687 INFO:teuthology.orchestra.run.smithi063.stdout:[{"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.688+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 msgr2=0x7fb98003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.688+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 0x7fb98003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb988006fd0 tx=0x7fb988006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.688+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 msgr2=0x7fb99c19a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.688+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb98c006010 tx=0x7fb98c005780 comp rx=0 tx=0).stop 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 shutdown_connections 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb980039650 0x7fb98003bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 --2- 172.21.15.63:0/3104842323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb99c1084e0 0x7fb99c19a0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:38.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 >> 172.21.15.63:0/3104842323 conn(0x7fb99c0fdf80 msgr2=0x7fb99c109f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:38.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 shutdown_connections 2024-04-25T18:34:38.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:38.689+0000 7fb9a2800700 1 -- 172.21.15.63:0/3104842323 wait complete. 2024-04-25T18:34:39.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:38 smithi063 ceph-mon[28006]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:39.089 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:39.532 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T18:34:39.533 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T18:34:39.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:39 smithi063 ceph-mon[28006]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:34:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:34:40.724 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:40 smithi063 ceph-mon[28006]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:42.002 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.002 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.002 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.002 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.003 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:34:42.003 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:42.003 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:34:42.003 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:41 smithi063 ceph-mon[28006]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 -- 172.21.15.63:0/2302126811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 msgr2=0x7fbd141088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 --2- 172.21.15.63:0/2302126811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd141088c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009910 tx=0x7fbd04009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 -- 172.21.15.63:0/2302126811 shutdown_connections 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 --2- 172.21.15.63:0/2302126811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd141088c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 -- 172.21.15.63:0/2302126811 >> 172.21.15.63:0/2302126811 conn(0x7fbd140fdf80 msgr2=0x7fbd141003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.312+0000 7fbd18e00700 1 -- 172.21.15.63:0/2302126811 shutdown_connections 2024-04-25T18:34:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.313+0000 7fbd18e00700 1 -- 172.21.15.63:0/2302126811 wait complete. 2024-04-25T18:34:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.314+0000 7fbd18e00700 1 Processor -- start 2024-04-25T18:34:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.314+0000 7fbd18e00700 1 -- start start 2024-04-25T18:34:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.314+0000 7fbd18e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.314+0000 7fbd18e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1410c940 con 0x7fbd141084e0 2024-04-25T18:34:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.315+0000 7fbd11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.315+0000 7fbd11e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45248/0 (socket says 172.21.15.63:45248) 2024-04-25T18:34:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.315+0000 7fbd11e00700 1 -- 172.21.15.63:0/1009837243 learned_addr learned my addr 172.21.15.63:0/1009837243 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.315+0000 7fbd11e00700 1 -- 172.21.15.63:0/1009837243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd141983f0 con 0x7fbd141084e0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.315+0000 7fbd11e00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd040056d0 tx=0x7fbd040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.316+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd04026070 con 0x7fbd141084e0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.316+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd0400bba0 con 0x7fbd141084e0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.316+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd04003920 con 0x7fbd141084e0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.316+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd141986e0 con 0x7fbd141084e0 2024-04-25T18:34:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.316+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1419c780 con 0x7fbd141084e0 2024-04-25T18:34:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.317+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fbd04029070 con 0x7fbd141084e0 2024-04-25T18:34:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.317+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd14103b90 con 0x7fbd141084e0 2024-04-25T18:34:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.318+0000 7fbd0aa00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 0x7fbcf803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.318+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbd0401e070 con 0x7fbd141084e0 2024-04-25T18:34:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.318+0000 7fbd11400700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 0x7fbcf803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.319+0000 7fbd11400700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 0x7fbcf803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd00006fd0 tx=0x7fbd00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.323+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd04013070 con 0x7fbd141084e0 2024-04-25T18:34:42.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:42.561+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbd14199210 con 0x7fbd141084e0 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:42 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1009837243' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T18:34:43.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.077+0000 7fbd0aa00700 1 -- 172.21.15.63:0/1009837243 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fbd04003b60 con 0x7fbd141084e0 2024-04-25T18:34:43.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 msgr2=0x7fbcf803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:43.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 0x7fbcf803bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbd00006fd0 tx=0x7fbd00006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 msgr2=0x7fbd14197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd040056d0 tx=0x7fbd040057b0 comp rx=0 tx=0).stop 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 shutdown_connections 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbcf80396b0 0x7fbcf803bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 --2- 172.21.15.63:0/1009837243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd141084e0 0x7fbd14197eb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.081+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 >> 172.21.15.63:0/1009837243 conn(0x7fbd140fdf80 msgr2=0x7fbd1410a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.082+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 shutdown_connections 2024-04-25T18:34:43.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:34:43.082+0000 7fbd18e00700 1 -- 172.21.15.63:0/1009837243 wait complete. 2024-04-25T18:34:43.084 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-04-25T18:34:43.923 INFO:tasks.cephadm:Adding mon.a on smithi063 2024-04-25T18:34:43.923 INFO:tasks.cephadm:Adding mon.b on smithi070 2024-04-25T18:34:43.924 INFO:tasks.cephadm:Adding mon.c on smithi114 2024-04-25T18:34:43.924 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c' 2024-04-25T18:34:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:44 smithi063 ceph-mon[28006]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:44 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1009837243' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T18:34:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:44 smithi063 ceph-mon[28006]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:34:44.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:34:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.601+0000 7f9e6afad700 1 -- 172.21.15.114:0/3508994085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 msgr2=0x7f9e641088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3508994085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e641088c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009910 tx=0x7f9e60009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 -- 172.21.15.114:0/3508994085 shutdown_connections 2024-04-25T18:34:45.603 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3508994085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e641088c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:45.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 -- 172.21.15.114:0/3508994085 >> 172.21.15.114:0/3508994085 conn(0x7f9e640fdf80 msgr2=0x7f9e641003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:45.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 -- 172.21.15.114:0/3508994085 shutdown_connections 2024-04-25T18:34:45.604 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 -- 172.21.15.114:0/3508994085 wait complete. 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 Processor -- start 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.602+0000 7f9e6afad700 1 -- start start 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.603+0000 7f9e6afad700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.603+0000 7f9e6afad700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6410c940 con 0x7f9e641084e0 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.603+0000 7f9e68d49700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.603+0000 7f9e68d49700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.114:58606/0 (socket says 172.21.15.114:58606) 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.603+0000 7f9e68d49700 1 -- 172.21.15.114:0/3957406443 learned_addr learned my addr 172.21.15.114:0/3957406443 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:34:45.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.604+0000 7f9e68d49700 1 -- 172.21.15.114:0/3957406443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e641983f0 con 0x7f9e641084e0 2024-04-25T18:34:45.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.604+0000 7f9e68d49700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9e600056d0 tx=0x7f9e600057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:45.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.604+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e60026070 con 0x7f9e641084e0 2024-04-25T18:34:45.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.604+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e641986e0 con 0x7f9e641084e0 2024-04-25T18:34:45.606 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.605+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6419c780 con 0x7f9e641084e0 2024-04-25T18:34:45.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.605+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e6000bba0 con 0x7f9e641084e0 2024-04-25T18:34:45.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.605+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e600037f0 con 0x7f9e641084e0 2024-04-25T18:34:45.607 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.605+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e64103b90 con 0x7f9e641084e0 2024-04-25T18:34:45.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.606+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9e60029070 con 0x7f9e641084e0 2024-04-25T18:34:45.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.606+0000 7f9e59ffb700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 0x7f9e4c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:45.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.606+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9e6001e070 con 0x7f9e641084e0 2024-04-25T18:34:45.608 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.607+0000 7f9e5bfff700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 0x7f9e4c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:45.609 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.608+0000 7f9e5bfff700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 0x7f9e4c03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e54006fd0 tx=0x7f9e54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:45.625 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.624+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e60013070 con 0x7f9e641084e0 2024-04-25T18:34:45.835 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.833+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c", "target": ["mon-mgr", ""]}) -- 0x7f9e64199210 con 0x7f9e4c039650 2024-04-25T18:34:45.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.849+0000 7f9e59ffb700 1 -- 172.21.15.114:0/3957406443 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9e64199210 con 0x7f9e4c039650 2024-04-25T18:34:45.850 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mon update... 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 msgr2=0x7f9e4c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 0x7f9e4c03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9e54006fd0 tx=0x7f9e54006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 msgr2=0x7f9e64197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9e600056d0 tx=0x7f9e600057b0 comp rx=0 tx=0).stop 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 shutdown_connections 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9e4c039650 0x7f9e4c03bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 --2- 172.21.15.114:0/3957406443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e641084e0 0x7f9e64197eb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 >> 172.21.15.114:0/3957406443 conn(0x7f9e640fdf80 msgr2=0x7f9e6410a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 shutdown_connections 2024-04-25T18:34:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:45.851+0000 7f9e6afad700 1 -- 172.21.15.114:0/3957406443 wait complete. 2024-04-25T18:34:46.491 DEBUG:teuthology.orchestra.run.smithi070:mon.b> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T18:34:46.493 DEBUG:teuthology.orchestra.run.smithi114:mon.c> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T18:34:46.496 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T18:34:46.496 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T18:34:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:34:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: Saving service mon spec with placement smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c;count:3 2024-04-25T18:34:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:34:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:47.152 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:34:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:47 smithi063 ceph-mon[28006]: Deploying daemon mon.c on smithi114 2024-04-25T18:34:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:48 smithi063 ceph-mon[28006]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:50.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- 172.21.15.114:0/634511649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 msgr2=0x7f4bb8101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:50.007 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 --2- 172.21.15.114:0/634511649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8101970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009910 tx=0x7f4ba8009c20 comp rx=0 tx=0).stop 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- 172.21.15.114:0/634511649 shutdown_connections 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 --2- 172.21.15.114:0/634511649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8101970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- 172.21.15.114:0/634511649 >> 172.21.15.114:0/634511649 conn(0x7f4bb80fce40 msgr2=0x7f4bb80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- 172.21.15.114:0/634511649 shutdown_connections 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- 172.21.15.114:0/634511649 wait complete. 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 Processor -- start 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.005+0000 7f4bbc937700 1 -- start start 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.006+0000 7f4bbc937700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.006+0000 7f4bbc937700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb810b810 con 0x7f4bb8101590 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.006+0000 7f4bb659c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.006+0000 7f4bb659c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.114:38234/0 (socket says 172.21.15.114:38234) 2024-04-25T18:34:50.008 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.006+0000 7f4bb659c700 1 -- 172.21.15.114:0/956735975 learned_addr learned my addr 172.21.15.114:0/956735975 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:34:50.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4bb659c700 1 -- 172.21.15.114:0/956735975 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb8199230 con 0x7f4bb8101590 2024-04-25T18:34:50.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4bb659c700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80056d0 tx=0x7f4ba80057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:50.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8026070 con 0x7f4bb8101590 2024-04-25T18:34:50.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb8199520 con 0x7f4bb8101590 2024-04-25T18:34:50.009 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba800bba0 con 0x7f4bb8101590 2024-04-25T18:34:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.007+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb81999e0 con 0x7f4bb8101590 2024-04-25T18:34:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.008+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ba8030060 con 0x7f4bb8101590 2024-04-25T18:34:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.009+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb8102a70 con 0x7f4bb8101590 2024-04-25T18:34:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.009+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4ba8029070 con 0x7f4bb8101590 2024-04-25T18:34:50.010 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.009+0000 7f4ba77fe700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 0x7f4ba003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:50.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.009+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4ba801e070 con 0x7f4bb8101590 2024-04-25T18:34:50.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.010+0000 7f4bb5d9b700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 0x7f4ba003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:50.011 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.010+0000 7f4bb5d9b700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 0x7f4ba003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4bac006fd0 tx=0x7f4bac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:50.027 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.025+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ba8013070 con 0x7f4bb8101590 2024-04-25T18:34:50.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.332+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4bb80640a0 con 0x7f4bb8101590 2024-04-25T18:34:50.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.333+0000 7f4ba77fe700 1 -- 172.21.15.114:0/956735975 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f4ba8004390 con 0x7f4bb8101590 2024-04-25T18:34:50.335 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:34:50.335 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","modified":"2024-04-25T18:32:36.435483Z","created":"2024-04-25T18:32:36.435483Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T18:34:50.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.335+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 msgr2=0x7f4ba003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:50.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.335+0000 7f4bbc937700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 0x7f4ba003bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4bac006fd0 tx=0x7f4bac006e40 comp rx=0 tx=0).stop 2024-04-25T18:34:50.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 msgr2=0x7f4bb8198cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:50.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80056d0 tx=0x7f4ba80057b0 comp rx=0 tx=0).stop 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 shutdown_connections 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba0039650 0x7f4ba003bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 --2- 172.21.15.114:0/956735975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bb8101590 0x7f4bb8198cf0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 >> 172.21.15.114:0/956735975 conn(0x7f4bb80fce40 msgr2=0x7f4bb8102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 shutdown_connections 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:50.336+0000 7f4bbc937700 1 -- 172.21.15.114:0/956735975 wait complete. 2024-04-25T18:34:50.338 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 1 2024-04-25T18:34:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:50 smithi063 ceph-mon[28006]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:50 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/956735975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:34:51.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:51 smithi114 systemd[1]: Started Ceph mon.c for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:34:51.976 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T18:34:51.976 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T18:34:52.212 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:34:56.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 -- 172.21.15.114:0/3229780933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 msgr2=0x7f5078005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 --2- 172.21.15.114:0/3229780933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f5078005610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f507c005fa0 tx=0x7f507c00ffc0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 -- 172.21.15.114:0/3229780933 shutdown_connections 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 --2- 172.21.15.114:0/3229780933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f5078005610 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 -- 172.21.15.114:0/3229780933 >> 172.21.15.114:0/3229780933 conn(0x7f508c0fcf30 msgr2=0x7f508c0ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 -- 172.21.15.114:0/3229780933 shutdown_connections 2024-04-25T18:34:56.585 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.584+0000 7f5090eee700 1 -- 172.21.15.114:0/3229780933 wait complete. 2024-04-25T18:34:56.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 Processor -- start 2024-04-25T18:34:56.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 -- start start 2024-04-25T18:34:56.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f508c196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:56.586 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f508c109390 con 0x7f508c1074d0 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5090eee700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f508c109210 con 0x7f508c197190 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:49224/0 (socket says 172.21.15.114:49224) 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 -- 172.21.15.114:0/359880103 learned_addr learned my addr 172.21.15.114:0/359880103 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 -- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 msgr2=0x7f508c197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 -- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 msgr2=0x7f508c197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.585+0000 7f5081d9b700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:34:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.586+0000 7f508a59c700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f508c196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:56.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.586+0000 7f508a59c700 1 -- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 msgr2=0x7f508c197610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:34:56.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.586+0000 7f508a59c700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.586+0000 7f508a59c700 1 -- 172.21.15.114:0/359880103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c06ba50 con 0x7f508c1074d0 2024-04-25T18:34:56.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f508a59c700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f508c196c50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f507c0137f0 tx=0x7f507c004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:56.588 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f507c005370 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c1a42c0 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507c005510 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c1a4770 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.587+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f507c011610 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.588+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508c102b80 con 0x7f508c1074d0 2024-04-25T18:34:56.589 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.589+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f507c040910 con 0x7f508c1074d0 2024-04-25T18:34:56.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.589+0000 7f5083fff700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 0x7f506c03b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:34:56.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.589+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f507c04ed10 con 0x7f508c1074d0 2024-04-25T18:34:56.590 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.589+0000 7f5081d9b700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 0x7f506c03b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:34:56.591 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.591+0000 7f5081d9b700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 0x7f506c03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5074009180 tx=0x7f5074008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:34:56.605 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.605+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f507c022070 con 0x7f508c1074d0 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: mon.a calling monitor election 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: mon.c calling monitor election 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: monmap epoch 2 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: last_changed 2024-04-25T18:34:51.398523+0000 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: min_mon_release 19 (squid) 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: election_strategy: 1 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: fsmap 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: mgrmap e13: x(active, since 63s) 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:56.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: mon.a calling monitor election 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: mon.c calling monitor election 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: monmap epoch 2 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: last_changed 2024-04-25T18:34:51.398523+0000 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: min_mon_release 19 (squid) 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: election_strategy: 1 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: fsmap 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: mgrmap e13: x(active, since 63s) 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:56.887 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.886+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f508c02df80 con 0x7f508c1074d0 2024-04-25T18:34:56.888 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.887+0000 7f5083fff700 1 -- 172.21.15.114:0/359880103 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f507c00b2c0 con 0x7f508c1074d0 2024-04-25T18:34:56.888 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:34:56.888 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":2,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","modified":"2024-04-25T18:34:51.398523Z","created":"2024-04-25T18:32:36.435483Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.889+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 msgr2=0x7f506c03b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.889+0000 7f5090eee700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 0x7f506c03b8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5074009180 tx=0x7f5074008040 comp rx=0 tx=0).stop 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 msgr2=0x7f508c196c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f508c196c50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f507c0137f0 tx=0x7f507c004d60 comp rx=0 tx=0).stop 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 shutdown_connections 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f506c0393f0 0x7f506c03b8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f508c1074d0 0x7f508c196c50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.890 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 --2- 172.21.15.114:0/359880103 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f508c197190 0x7f508c197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:34:56.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 >> 172.21.15.114:0/359880103 conn(0x7f508c0fcf30 msgr2=0x7f508c0feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:34:56.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 shutdown_connections 2024-04-25T18:34:56.891 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:34:56.890+0000 7f5090eee700 1 -- 172.21.15.114:0/359880103 wait complete. 2024-04-25T18:34:56.891 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 2 2024-04-25T18:34:57.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:34:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: Deploying daemon mon.b on smithi070 2024-04-25T18:34:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/359880103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:34:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: Deploying daemon mon.b on smithi070 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/359880103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:34:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:34:58.630 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T18:34:58.631 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T18:34:58.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:34:58 smithi114 ceph-mon[32988]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:34:58.870 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:34:58.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:34:58 smithi063 ceph-mon[28006]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.163 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 podman[32802]: 2024-04-25 18:35:00.005642497 +0000 UTC m=+0.018649129 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 -- 172.21.15.114:0/143785958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 msgr2=0x7f0d98106880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/143785958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98106880 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009910 tx=0x7f0d88009c20 comp rx=0 tx=0).stop 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 -- 172.21.15.114:0/143785958 shutdown_connections 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/143785958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98106880 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 -- 172.21.15.114:0/143785958 >> 172.21.15.114:0/143785958 conn(0x7f0d980767a0 msgr2=0x7f0d98076bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 -- 172.21.15.114:0/143785958 shutdown_connections 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.245+0000 7f0d9e01e700 1 -- 172.21.15.114:0/143785958 wait complete. 2024-04-25T18:35:00.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 Processor -- start 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 -- start start 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9810b960 con 0x7f0d981064a0 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d9e01e700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9810b7e0 con 0x7f0d98198670 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:41568/0 (socket says 172.21.15.114:41568) 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 -- 172.21.15.114:0/914224098 learned_addr learned my addr 172.21.15.114:0/914224098 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 -- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 msgr2=0x7f0d9819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 -- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 msgr2=0x7f0d9819c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T18:35:00.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.246+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.247+0000 7f0d977fe700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.247+0000 7f0d977fe700 1 -- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 msgr2=0x7f0d9819c9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.247+0000 7f0d977fe700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.247+0000 7f0d977fe700 1 -- 172.21.15.114:0/914224098 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d98198cb0 con 0x7f0d981064a0 2024-04-25T18:35:00.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d977fe700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98198130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d88014000 tx=0x7f0d88005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0d88005dc0 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9819cef0 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9819d470 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d88004710 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.248+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f0d8800bcf0 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.249+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d84005330 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.249+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0d88029070 con 0x7f0d981064a0 2024-04-25T18:35:00.250 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.250+0000 7f0d94ff9700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 0x7f0d8003b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:00.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.250+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0d8801e070 con 0x7f0d981064a0 2024-04-25T18:35:00.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.250+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 0x7f0d8003b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:00.251 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.251+0000 7f0d96ffd700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 0x7f0d8003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009180 tx=0x7f0d8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:00.267 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.267+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d88013070 con 0x7f0d981064a0 2024-04-25T18:35:00.300 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.299+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800e610 con 0x7f0d981064a0 2024-04-25T18:35:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 podman[32802]: 2024-04-25 18:35:00.233418945 +0000 UTC m=+0.246425581 container init 229d68a239fe78ba4a5f3293037c1d4f0119695a4c72a0c8b43336b8fd22f949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:35:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 podman[32802]: 2024-04-25 18:35:00.236424757 +0000 UTC m=+0.249431389 container start 229d68a239fe78ba4a5f3293037c1d4f0119695a4c72a0c8b43336b8fd22f949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-25T18:35:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T18:35:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: ceph version 19.0.0-3156-gd69f13d8 (d69f13d827952bb07bb9b7973bca14f97d764d6f) squid (dev), process ceph-mon, pid 2 2024-04-25T18:35:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pidfile_write: ignore empty --pid-file 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: load: jerasure load: lrc 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: RocksDB version: 7.9.2 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Git sha 0 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Compile date 2024-04-22 10:57:25 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: DB SUMMARY 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: DB Session ID: NDWCGKZXAFG0JSWWQ4MU 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: CURRENT file: CURRENT 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T18:35:00.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.error_if_exists: 0 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.create_if_missing: 0 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.paranoid_checks: 1 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.env: 0x558407550cc0 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T18:35:00.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.info_log: 0x558408b09be0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.statistics: (nil) 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.use_fsync: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_log_file_size: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_fallocate: 1 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.use_direct_reads: 0 2024-04-25T18:35:00.457 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.db_log_dir: 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.wal_dir: 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.write_buffer_manager: 0x558408a9d0e0 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T18:35:00.458 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.unordered_write: 0 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.row_cache: None 2024-04-25T18:35:00.459 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.wal_filter: None 2024-04-25T18:35:00.461 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T18:35:00.461 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T18:35:00.461 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.two_write_queues: 0 2024-04-25T18:35:00.461 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.wal_compression: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.atomic_flush: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.log_readahead_size: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T18:35:00.462 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_background_jobs: 2 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_background_compactions: -1 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_subcompactions: 1 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_open_files: -1 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T18:35:00.463 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_background_flushes: -1 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Compression algorithms supported: 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kZSTD supported: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kXpressCompression supported: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kZlibCompression supported: 1 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kSnappyCompression supported: 1 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kLZ4Compression supported: 1 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: kBZip2Compression supported: 0 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T18:35:00.464 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.merge_operator: 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_filter: None 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558408b09dc0) 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: cache_index_and_filter_blocks: 1 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: pin_top_level_index_and_filter: 1 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_type: 0 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: data_block_index_type: 0 2024-04-25T18:35:00.465 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_shortening: 1 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: checksum: 4 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: no_block_cache: 0 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache: 0x558408acf1f0 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_name: BinnedLRUCache 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_options: 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: capacity : 536870912 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: num_shard_bits : 4 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: strict_capacity_limit : 0 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: high_pri_pool_ratio: 0.000 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_cache_compressed: (nil) 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: persistent_cache: (nil) 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_size: 4096 2024-04-25T18:35:00.466 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_size_deviation: 10 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_restart_interval: 16 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: index_block_restart_interval: 1 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: metadata_block_size: 4096 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: partition_filters: 0 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: use_delta_encoding: 1 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: filter_policy: bloomfilter 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: whole_key_filtering: 1 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: verify_compression: 0 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: read_amp_bytes_per_bit: 0 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: format_version: 5 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: enable_index_compression: 1 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: block_align: 0 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: max_auto_readahead_size: 262144 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: prepopulate_block_cache: 0 2024-04-25T18:35:00.467 INFO:journalctl@ceph.mon.b.smithi070.stdout: initial_auto_readahead_size: 8192 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression: NoCompression 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.num_levels: 7 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T18:35:00.468 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T18:35:00.469 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T18:35:00.470 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T18:35:00.471 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.inplace_update_support: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.bloom_locality: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.max_successive_merges: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T18:35:00.472 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.ttl: 2592000 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enable_blob_files: false 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.min_blob_size: 0 2024-04-25T18:35:00.473 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5b5cb46c-3b95-4ca5-8df2-1c9d3f70e5a2 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714070100265834, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T18:35:00.474 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714070100266238, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714070100, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5b5cb46c-3b95-4ca5-8df2-1c9d3f70e5a2", "db_session_id": "NDWCGKZXAFG0JSWWQ4MU", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714070100266321, "job": 1, "event": "recovery_finished"} 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558408be4000 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: DB pointer 0x558408bce000 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** DB Stats ** 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T18:35:00.475 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T18:35:00.476 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** Compaction Stats [default] ** 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T18:35:00.477 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: Block cache BinnedLRUCache@0x558408acf1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T18:35:00.478 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: using public_addr v2:172.21.15.70:0/0 -> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: starting mon.b rank -1 at public addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] at bind addrs [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(???) e0 preinit fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout: e1 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout: legacy client fscid: -1 2024-04-25T18:35:00.479 INFO:journalctl@ceph.mon.b.smithi070.stdout: 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout: No filesystems configured 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.480 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Added host smithi070 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.481 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Deploying cephadm binary to smithi114 2024-04-25T18:35:00.482 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Added host smithi114 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.483 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:35:00.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.484 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.485 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1009837243' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1009837243' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.486 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Saving service mon spec with placement smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c;count:3 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Deploying daemon mon.c on smithi114 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.487 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/956735975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.a calling monitor election 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.c calling monitor election 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.488 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: monmap epoch 2 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: last_changed 2024-04-25T18:34:51.398523+0000 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: min_mon_release 19 (squid) 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: election_strategy: 1 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: fsmap 2024-04-25T18:35:00.489 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mgrmap e13: x(active, since 63s) 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: Deploying daemon mon.b on smithi070 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/359880103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:00.490 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:00.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 ceph-mon[32820]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T18:35:00.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 bash[32802]: 229d68a239fe78ba4a5f3293037c1d4f0119695a4c72a0c8b43336b8fd22f949 2024-04-25T18:35:00.491 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:00 smithi070 systemd[1]: Started Ceph mon.b for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:35:00.547 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:00.546+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0d840058f0 con 0x7f0d981064a0 2024-04-25T18:35:05.320 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.318+0000 7f0d94ff9700 1 -- 172.21.15.114:0/914224098 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f0d88038510 con 0x7f0d981064a0 2024-04-25T18:35:05.321 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-04-25T18:35:05.321 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":3,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","modified":"2024-04-25T18:35:00.296873Z","created":"2024-04-25T18:32:36.435483Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T18:35:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.321+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 msgr2=0x7f0d8003b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.321+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 0x7f0d8003b900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009180 tx=0x7f0d8c008040 comp rx=0 tx=0).stop 2024-04-25T18:35:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.321+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 msgr2=0x7f0d98198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.321+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98198130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d88014000 tx=0x7f0d88005030 comp rx=0 tx=0).stop 2024-04-25T18:35:05.323 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 shutdown_connections 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0d80039440 0x7f0d8003b900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d981064a0 0x7f0d98198130 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 --2- 172.21.15.114:0/914224098 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0d98198670 0x7f0d9819c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 >> 172.21.15.114:0/914224098 conn(0x7f0d980767a0 msgr2=0x7f0d98104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 shutdown_connections 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:05.322+0000 7f0d9e01e700 1 -- 172.21.15.114:0/914224098 wait complete. 2024-04-25T18:35:05.324 INFO:teuthology.orchestra.run.smithi114.stderr:dumped monmap epoch 3 2024-04-25T18:35:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: mon.c calling monitor election 2024-04-25T18:35:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:35:05.622 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: mon.a calling monitor election 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/914224098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: monmap epoch 3 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: min_mon_release 19 (squid) 2024-04-25T18:35:05.623 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: election_strategy: 1 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: fsmap 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: mgrmap e13: x(active, since 72s) 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.624 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: mon.c calling monitor election 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: mon.a calling monitor election 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/914224098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:05.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: monmap epoch 3 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: min_mon_release 19 (squid) 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: election_strategy: 1 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: fsmap 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: mgrmap e13: x(active, since 72s) 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T18:35:05.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:05.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:06.000 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T18:35:06.001 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T18:35:06.438 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:35:06.612 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:06 smithi114 ceph-mon[32988]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:06.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:06.613 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:06.759 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:06 smithi063 ceph-mon[28006]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:06.759 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:06.759 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.c calling monitor election 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.a calling monitor election 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/914224098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: monmap epoch 3 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: min_mon_release 19 (squid) 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: election_strategy: 1 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:07.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: fsmap 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mgrmap e13: x(active, since 72s) 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:07.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:08.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.b calling monitor election 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.b calling monitor election 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.c calling monitor election 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.a calling monitor election 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T18:35:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: monmap epoch 3 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: min_mon_release 19 (squid) 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: election_strategy: 1 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: fsmap 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: mgrmap e13: x(active, since 74s) 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:07 smithi070 ceph-mon[32820]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.214 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mon.b calling monitor election 2024-04-25T18:35:08.214 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:35:08.214 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:35:08.214 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mon.b calling monitor election 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mon.c calling monitor election 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mon.a calling monitor election 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: monmap epoch 3 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: min_mon_release 19 (squid) 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: election_strategy: 1 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:08.215 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: fsmap 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: mgrmap e13: x(active, since 74s) 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.216 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:07 smithi114 ceph-mon[32988]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mon.b calling monitor election 2024-04-25T18:35:08.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:35:08.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mon.b calling monitor election 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mon.c calling monitor election 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mon.a calling monitor election 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: monmap epoch 3 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: last_changed 2024-04-25T18:35:00.296873+0000 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: created 2024-04-25T18:32:36.435483+0000 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: min_mon_release 19 (squid) 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: election_strategy: 1 2024-04-25T18:35:08.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: 0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: 1: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.c 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: fsmap 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: mgrmap e13: x(active, since 74s) 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:08.302 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:07 smithi063 ceph-mon[28006]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:35:08.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 -- 172.21.15.63:0/2294532552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c01015f0 msgr2=0x7f62c01019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 --2- 172.21.15.63:0/2294532552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c01015f0 0x7f62c01019d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009910 tx=0x7f62b0009c20 comp rx=0 tx=0).stop 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 -- 172.21.15.63:0/2294532552 shutdown_connections 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 --2- 172.21.15.63:0/2294532552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c01015f0 0x7f62c01019d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 -- 172.21.15.63:0/2294532552 >> 172.21.15.63:0/2294532552 conn(0x7f62c0076ea0 msgr2=0x7f62c00772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 -- 172.21.15.63:0/2294532552 shutdown_connections 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.906+0000 7f62c6600700 1 -- 172.21.15.63:0/2294532552 wait complete. 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.907+0000 7f62c6600700 1 Processor -- start 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 -- start start 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 0x7f62c019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 0x7f62c019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:08.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c019f110 0x7f62c019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010b820 con 0x7f62c019a890 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010b6a0 con 0x7f62c01015f0 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62c6600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62c010b9a0 con 0x7f62c019f110 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.908+0000 7f62bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 0x7f62c019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 0x7f62c019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 0x7f62c019ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34012/0 (socket says 172.21.15.63:34012) 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bea00700 1 -- 172.21.15.63:0/2043791498 learned_addr learned my addr 172.21.15.63:0/2043791498 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bfe00700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c019f110 0x7f62c019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 -- 172.21.15.63:0/2043791498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c019f110 msgr2=0x7f62c019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c019f110 0x7f62c019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 -- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 msgr2=0x7f62c019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 0x7f62c019ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 -- 172.21.15.63:0/2043791498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c01a6e80 con 0x7f62c01015f0 2024-04-25T18:35:08.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.909+0000 7f62bf400700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 0x7f62c019a350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009910 tx=0x7f62b0004f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.910+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0026070 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.910+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62b000b6f0 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.910+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c01a70b0 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.910+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c01a7640 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.910+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009e60 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.911+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c00640a0 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.911+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f62b0029070 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.912+0000 7f62b7e00700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 0x7f62a403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.912+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f62b004d320 con 0x7f62c01015f0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.912+0000 7f62bea00700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 0x7f62a403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.913+0000 7f62bea00700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 0x7f62a403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009b80 tx=0x7f62ac0092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:08.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:08.918+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f62b0013070 con 0x7f62c01015f0 2024-04-25T18:35:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.153+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f62c01a78f0 con 0x7f62c01015f0 2024-04-25T18:35:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.153+0000 7f62b7e00700 1 -- 172.21.15.63:0/2043791498 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f62b0029360 con 0x7f62c01015f0 2024-04-25T18:35:09.154 INFO:teuthology.orchestra.run.smithi063.stdout:# minimal ceph.conf for e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:09.155 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-04-25T18:35:09.155 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:35:09.155 INFO:teuthology.orchestra.run.smithi063.stdout: mon_host = [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2024-04-25T18:35:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 msgr2=0x7f62a403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 0x7f62a403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009b80 tx=0x7f62ac0092c0 comp rx=0 tx=0).stop 2024-04-25T18:35:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 msgr2=0x7f62c019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 0x7f62c019a350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009910 tx=0x7f62b0004f80 comp rx=0 tx=0).stop 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.158 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T18:35:09.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:09.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:09.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:09.159 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:09 smithi063 ceph-mon[28006]: Reconfiguring daemon mon.a on smithi063 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 shutdown_connections 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62a4039ae0 0x7f62a403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f62c019f110 0x7f62c019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62c01015f0 0x7f62c019a350 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 --2- 172.21.15.63:0/2043791498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62c019a890 0x7f62c019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 >> 172.21.15.63:0/2043791498 conn(0x7f62c0076ea0 msgr2=0x7f62c0104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.156+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 shutdown_connections 2024-04-25T18:35:09.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:09.157+0000 7f62c6600700 1 -- 172.21.15.63:0/2043791498 wait complete. 2024-04-25T18:35:09.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:09 smithi114 ceph-mon[32988]: Reconfiguring daemon mon.a on smithi063 2024-04-25T18:35:09.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:35:09.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:09.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:09 smithi070 ceph-mon[32820]: Reconfiguring daemon mon.a on smithi063 2024-04-25T18:35:09.973 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T18:35:09.974 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:35:09.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T18:35:10.064 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:35:10.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:35:10.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:10 smithi063 ceph-mon[28006]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:10.182 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:10 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2043791498' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:10.194 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:35:10.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T18:35:10.226 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:35:10.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:35:10.293 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:35:10.293 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T18:35:10.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:10 smithi114 ceph-mon[32988]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:10.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:10 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2043791498' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:10.324 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:35:10.324 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:35:10.390 INFO:tasks.cephadm:Adding mgr.x on smithi063 2024-04-25T18:35:10.391 INFO:tasks.cephadm:Adding mgr.y on smithi070 2024-04-25T18:35:10.391 INFO:tasks.cephadm:Adding mgr.z on smithi114 2024-04-25T18:35:10.391 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi063=x;smithi070=y;smithi114=z' 2024-04-25T18:35:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:10 smithi070 ceph-mon[32820]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:10 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2043791498' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:10.664 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 -- 172.21.15.114:0/434536856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd41025d0 msgr2=0x7f0dd41029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 --2- 172.21.15.114:0/434536856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd41025d0 0x7f0dd41029f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009910 tx=0x7f0dc4009c20 comp rx=0 tx=0).stop 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 -- 172.21.15.114:0/434536856 shutdown_connections 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 --2- 172.21.15.114:0/434536856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd41025d0 0x7f0dd41029f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 -- 172.21.15.114:0/434536856 >> 172.21.15.114:0/434536856 conn(0x7f0dd40fdb70 msgr2=0x7f0dd40fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 -- 172.21.15.114:0/434536856 shutdown_connections 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 -- 172.21.15.114:0/434536856 wait complete. 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.197+0000 7f0dd9198700 1 Processor -- start 2024-04-25T18:35:12.199 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 -- start start 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 0x7f0dd4197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd4198270 0x7f0dd419d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd4108510 con 0x7f0dd4198270 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd4108390 con 0x7f0dd41986f0 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd9198700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd4108690 con 0x7f0dd41025d0 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.114:41410/0 (socket says 172.21.15.114:41410) 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 -- 172.21.15.114:0/3756844239 learned_addr learned my addr 172.21.15.114:0/3756844239 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd2d9d700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 0x7f0dd4197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd259c700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd4198270 0x7f0dd419d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 -- 172.21.15.114:0/3756844239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 msgr2=0x7f0dd4197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:12.202 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 0x7f0dd4197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 -- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd4198270 msgr2=0x7f0dd419d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd4198270 0x7f0dd419d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd359e700 1 -- 172.21.15.114:0/3756844239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd41a0150 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.198+0000 7f0dd2d9d700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 0x7f0dd4197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dd359e700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00ba50 tx=0x7f0dcc00bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc019070 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dcc014440 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc00ce60 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd41a0440 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.199+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd41a09b0 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.200+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0dcc01c070 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.200+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd40640a0 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.200+0000 7f0dbbfff700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 0x7f0dbc03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.200+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0dcc04bf10 con 0x7f0dd41986f0 2024-04-25T18:35:12.203 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.200+0000 7f0dd2d9d700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 0x7f0dbc03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:12.204 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.201+0000 7f0dd2d9d700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 0x7f0dbc03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4199210 tx=0x7f0dc401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:12.206 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.205+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dcc012500 con 0x7f0dd41986f0 2024-04-25T18:35:12.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.416+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi063=x;smithi070=y;smithi114=z", "target": ["mon-mgr", ""]}) -- 0x7f0dd4062f10 con 0x7f0dbc039ae0 2024-04-25T18:35:12.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.429+0000 7f0dbbfff700 1 -- 172.21.15.114:0/3756844239 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0dd4062f10 con 0x7f0dbc039ae0 2024-04-25T18:35:12.431 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mgr update... 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 msgr2=0x7f0dbc03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 0x7f0dbc03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4199210 tx=0x7f0dc401a040 comp rx=0 tx=0).stop 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 msgr2=0x7f0dd419f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc00ba50 tx=0x7f0dcc00bd60 comp rx=0 tx=0).stop 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 shutdown_connections 2024-04-25T18:35:12.433 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0dbc039ae0 0x7f0dbc03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0dd41025d0 0x7f0dd4197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0dd41986f0 0x7f0dd419f850 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 --2- 172.21.15.114:0/3756844239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0dd4198270 0x7f0dd419d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 >> 172.21.15.114:0/3756844239 conn(0x7f0dd40fdb70 msgr2=0x7f0dd40fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 shutdown_connections 2024-04-25T18:35:12.434 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:35:12.432+0000 7f0dd9198700 1 -- 172.21.15.114:0/3756844239 wait complete. 2024-04-25T18:35:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: Reconfiguring mon.b (monmap changed)... 2024-04-25T18:35:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: Reconfiguring daemon mon.b on smithi070 2024-04-25T18:35:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: Reconfiguring mon.b (monmap changed)... 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: Reconfiguring daemon mon.b on smithi070 2024-04-25T18:35:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: Reconfiguring mon.b (monmap changed)... 2024-04-25T18:35:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: Reconfiguring daemon mon.b on smithi070 2024-04-25T18:35:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:12.976 DEBUG:teuthology.orchestra.run.smithi070:mgr.y> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T18:35:12.979 DEBUG:teuthology.orchestra.run.smithi114:mgr.z> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T18:35:12.981 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T18:35:12.981 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:35:12.981 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T18:35:13.041 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T18:35:13.041 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-04-25T18:35:13.097 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 18:34:11.670661003 +0000 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 18:28:05.560063917 +0000 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 18:28:05.560063917 +0000 2024-04-25T18:35:13.098 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 18:28:05.560063917 +0000 2024-04-25T18:35:13.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T18:35:13.179 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T18:35:13.180 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T18:35:13.180 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000395883 s, 1.3 MB/s 2024-04-25T18:35:13.184 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T18:35:13.243 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 18:34:11.752660212 +0000 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 18:28:06.177058017 +0000 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 18:28:06.177058017 +0000 2024-04-25T18:35:13.294 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 18:28:06.177058017 +0000 2024-04-25T18:35:13.295 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T18:35:13.378 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T18:35:13.378 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T18:35:13.378 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000369541 s, 1.4 MB/s 2024-04-25T18:35:13.380 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T18:35:13.441 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 18:34:11.829659469 +0000 2024-04-25T18:35:13.497 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 18:28:06.788052175 +0000 2024-04-25T18:35:13.498 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 18:28:06.788052175 +0000 2024-04-25T18:35:13.498 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 18:28:06.788052175 +0000 2024-04-25T18:35:13.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T18:35:13.581 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T18:35:13.581 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T18:35:13.581 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000360575 s, 1.4 MB/s 2024-04-25T18:35:13.586 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T18:35:13.646 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 18:34:11.901658775 +0000 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 18:28:07.362046687 +0000 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 18:28:07.362046687 +0000 2024-04-25T18:35:13.701 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 18:28:07.362046687 +0000 2024-04-25T18:35:13.702 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T18:35:13.785 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T18:35:13.785 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T18:35:13.785 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000363633 s, 1.4 MB/s 2024-04-25T18:35:13.790 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T18:35:13.853 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-25T18:35:13.853 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T18:35:13.871 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T18:35:13.871 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-25T18:35:13.928 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T18:35:13.928 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:13.928 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T18:35:13.928 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:13.929 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:13.929 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-25 18:34:59.811076952 +0000 2024-04-25T18:35:13.929 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-25 18:26:49.742634482 +0000 2024-04-25T18:35:13.929 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-25 18:26:49.742634482 +0000 2024-04-25T18:35:13.929 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-25 18:26:49.742634482 +0000 2024-04-25T18:35:13.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T18:35:13.994 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-25T18:35:13.994 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-25T18:35:13.994 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000124691 s, 4.1 MB/s 2024-04-25T18:35:13.998 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T18:35:14.056 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-25T18:35:14.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi063=x;smithi070=y;smithi114=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:14.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: Saving service mgr spec with placement smithi063=x;smithi070=y;smithi114=z;count:3 2024-04-25T18:35:14.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:14.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:14.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-25 18:34:59.811076952 +0000 2024-04-25T18:35:14.113 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-25 18:26:50.078624312 +0000 2024-04-25T18:35:14.114 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-25 18:26:50.078624312 +0000 2024-04-25T18:35:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-25 18:26:50.078624312 +0000 2024-04-25T18:35:14.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi063=x;smithi070=y;smithi114=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: Saving service mgr spec with placement smithi063=x;smithi070=y;smithi114=z;count:3 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:14.176 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi063=x;smithi070=y;smithi114=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: Saving service mgr spec with placement smithi063=x;smithi070=y;smithi114=z;count:3 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T18:35:14.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:14.179 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-25T18:35:14.179 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-25T18:35:14.179 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000122432 s, 4.2 MB/s 2024-04-25T18:35:14.180 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T18:35:14.238 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-25 18:34:59.811076952 +0000 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-25 18:26:50.406614383 +0000 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-25 18:26:50.406614383 +0000 2024-04-25T18:35:14.294 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-25 18:26:50.406614383 +0000 2024-04-25T18:35:14.294 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T18:35:14.357 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-25T18:35:14.357 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-25T18:35:14.357 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000114861 s, 4.5 MB/s 2024-04-25T18:35:14.358 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T18:35:14.415 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-25T18:35:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T18:35:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:14.471 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-25 18:34:59.812076923 +0000 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-25 18:26:50.723604788 +0000 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-25 18:26:50.723604788 +0000 2024-04-25T18:35:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-25 18:26:50.723604788 +0000 2024-04-25T18:35:14.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T18:35:14.534 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-25T18:35:14.534 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-25T18:35:14.535 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.8827e-05 s, 5.8 MB/s 2024-04-25T18:35:14.536 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T18:35:14.591 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-04-25T18:35:14.592 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T18:35:14.606 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T18:35:14.606 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-25 18:34:50.193306464 +0000 2024-04-25T18:35:14.660 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-25 18:26:49.751933777 +0000 2024-04-25T18:35:14.661 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-25 18:26:49.751933777 +0000 2024-04-25T18:35:14.661 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-25 18:26:49.751933777 +0000 2024-04-25T18:35:14.661 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T18:35:14.723 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-25T18:35:14.723 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-25T18:35:14.723 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000213603 s, 2.4 MB/s 2024-04-25T18:35:14.725 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T18:35:14.781 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-25 18:34:50.193306464 +0000 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-25 18:26:50.099926146 +0000 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-25 18:26:50.099926146 +0000 2024-04-25T18:35:14.836 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-25 18:26:50.099926146 +0000 2024-04-25T18:35:14.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T18:35:14.898 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-25T18:35:14.898 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-25T18:35:14.898 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00020605 s, 2.5 MB/s 2024-04-25T18:35:14.899 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T18:35:14.954 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-25 18:34:50.194306443 +0000 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-25 18:26:50.434918799 +0000 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-25 18:26:50.434918799 +0000 2024-04-25T18:35:15.009 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-25 18:26:50.434918799 +0000 2024-04-25T18:35:15.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T18:35:15.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: Reconfiguring mon.c (monmap changed)... 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: Reconfiguring daemon mon.c on smithi114 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-25T18:35:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-25T18:35:15.072 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000177537 s, 2.9 MB/s 2024-04-25T18:35:15.074 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: Reconfiguring mon.c (monmap changed)... 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: Reconfiguring daemon mon.c on smithi114 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:15.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.129 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-04-25 18:34:50.194306443 +0000 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-04-25 18:26:50.766911517 +0000 2024-04-25T18:35:15.183 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-04-25 18:26:50.766911517 +0000 2024-04-25T18:35:15.184 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-04-25 18:26:50.766911517 +0000 2024-04-25T18:35:15.184 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: Reconfiguring mon.c (monmap changed)... 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: Reconfiguring daemon mon.c on smithi114 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:15.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-04-25T18:35:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-04-25T18:35:15.247 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000176021 s, 2.9 MB/s 2024-04-25T18:35:15.248 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T18:35:15.303 INFO:tasks.cephadm:Deploying osd.0 on smithi063 with /dev/vg_nvme/lv_4... 2024-04-25T18:35:15.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T18:35:15.760 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:35:16.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:15 smithi063 ceph-mon[28006]: Deploying daemon mgr.z on smithi114 2024-04-25T18:35:16.160 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:15 smithi114 ceph-mon[32988]: Deploying daemon mgr.z on smithi114 2024-04-25T18:35:16.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:15 smithi070 ceph-mon[32820]: Deploying daemon mgr.z on smithi114 2024-04-25T18:35:16.816 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:16 smithi114 systemd[1]: Starting Ceph mgr.z for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:35:17.072 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:16 smithi114 ceph-mon[32988]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:17.073 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:16 smithi114 podman[34831]: 2024-04-25 18:35:16.724023253 +0000 UTC m=+0.017428331 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:35:17.073 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:16 smithi114 podman[34831]: 2024-04-25 18:35:16.837086857 +0000 UTC m=+0.130491965 container create cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T18:35:17.073 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 podman[34831]: 2024-04-25 18:35:17.072570723 +0000 UTC m=+0.365975785 container init cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:35:17.122 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:16 smithi070 ceph-mon[32820]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:17.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:16 smithi063 ceph-mon[28006]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:17.414 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 podman[34831]: 2024-04-25 18:35:17.075912021 +0000 UTC m=+0.369317084 container start cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T18:35:17.414 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-mgr[34850]: -- 172.21.15.114:0/4196200537 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610a70d85a0 con 0x5610a70ac800 2024-04-25T18:35:17.414 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 bash[34831]: cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 2024-04-25T18:35:17.414 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 systemd[1]: Started Ceph mgr.z for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:35:17.414 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:17.223+0000 7fe04613a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:35:17.415 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:17.413+0000 7fe04613a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:35:17.724 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:17.515+0000 7fe04613a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:35:18.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:17.841+0000 7fe04613a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:35:18.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:17.926+0000 7fe04613a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:35:18.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:18 smithi114 ceph-mon[32988]: Deploying daemon mgr.y on smithi070 2024-04-25T18:35:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:18.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:18 smithi063 ceph-mon[28006]: Deploying daemon mgr.y on smithi070 2024-04-25T18:35:18.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:18.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:18.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:18.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T18:35:18.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:18.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:18.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:18 smithi070 ceph-mon[32820]: Deploying daemon mgr.y on smithi070 2024-04-25T18:35:18.897 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T18:35:18.898 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T18:35:18.898 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T18:35:18.898 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T18:35:18.898 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0370751 s, 283 MB/s 2024-04-25T18:35:18.898 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T18:35:19.453 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 systemd[1]: Starting Ceph mgr.y for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:35:19.512 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:19.164+0000 7fe04613a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:35:19.512 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:19.337+0000 7fe04613a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:35:19.512 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:19.510+0000 7fe04613a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 podman[34193]: 2024-04-25 18:35:19.460137402 +0000 UTC m=+0.017704220 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 podman[34193]: 2024-04-25 18:35:19.557358068 +0000 UTC m=+0.114924862 container create d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 podman[34193]: 2024-04-25 18:35:19.761882493 +0000 UTC m=+0.319449303 container init d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 podman[34193]: 2024-04-25 18:35:19.764430385 +0000 UTC m=+0.321997190 container start d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 ceph-mgr[34212]: -- 172.21.15.70:0/3311450749 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555fae79e5a0 con 0x555fae773000 2024-04-25T18:35:19.829 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 bash[34193]: d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 2024-04-25T18:35:19.881 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-04-25T18:35:20.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:19.799+0000 7fe04613a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:35:20.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:19.891+0000 7fe04613a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:35:20.101 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 systemd[1]: Started Ceph mgr.y for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:35:20.101 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:19.916+0000 7f43674bf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:35:20.101 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:20.100+0000 7f43674bf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:35:20.326 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:20.201+0000 7f43674bf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:20.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:20.093+0000 7fe04613a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:35:20.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:20.293+0000 7fe04613a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:20.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:20.934 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:20.522+0000 7f43674bf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:35:20.935 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:20 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:20.607+0000 7f43674bf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:35:21.259 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:20.932+0000 7fe04613a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:35:21.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:21.366+0000 7fe04613a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:35:21.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:21.465+0000 7fe04613a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:35:22.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:21.671+0000 7fe04613a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:35:22.192 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:21 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:21.856+0000 7f43674bf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:35:22.193 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.025+0000 7f43674bf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:35:22.193 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.191+0000 7f43674bf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:35:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.531 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.195+0000 7fe04613a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:35:22.531 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.287+0000 7fe04613a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:22.532 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T18:35:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T18:35:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:22.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.472+0000 7f43674bf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:35:22.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.561+0000 7f43674bf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:35:22.760 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.759+0000 7f43674bf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:35:22.816 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.530+0000 7fe04613a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:35:22.816 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.661+0000 7fe04613a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:35:22.817 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.755+0000 7fe04613a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:35:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.948+0000 7ff3e2e00700 1 -- 172.21.15.63:0/3858848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc104570 msgr2=0x7ff3dc10ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:22.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.948+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/3858848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc104570 0x7ff3dc10ae20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400b0d0 tx=0x7ff3d400b3e0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 -- 172.21.15.63:0/3858848536 shutdown_connections 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/3858848536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc104570 0x7ff3dc10ae20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/3858848536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/3858848536 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc103bd0 0x7ff3dc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 -- 172.21.15.63:0/3858848536 >> 172.21.15.63:0/3858848536 conn(0x7ff3dc0fdf70 msgr2=0x7ff3dc1003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 -- 172.21.15.63:0/3858848536 shutdown_connections 2024-04-25T18:35:22.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.949+0000 7ff3e2e00700 1 -- 172.21.15.63:0/3858848536 wait complete. 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 Processor -- start 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 -- start start 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc103bd0 0x7ff3dc198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc104570 0x7ff3dc19ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc10e4a0 con 0x7ff3dc103bd0 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc10e320 con 0x7ff3dc1029d0 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.950+0000 7ff3e2e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff3dc10e620 con 0x7ff3dc104570 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:22.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:51180/0 (socket says 172.21.15.63:51180) 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 -- 172.21.15.63:0/2416359482 learned_addr learned my addr 172.21.15.63:0/2416359482 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1000700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc103bd0 0x7ff3dc198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e2400700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc104570 0x7ff3dc19ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 -- 172.21.15.63:0/2416359482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc104570 msgr2=0x7ff3dc19ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc104570 0x7ff3dc19ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 -- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc103bd0 msgr2=0x7ff3dc198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc103bd0 0x7ff3dc198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.951+0000 7ff3e1a00700 1 -- 172.21.15.63:0/2416359482 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3dc1a0680 con 0x7ff3dc1029d0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3e1a00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000e970 tx=0x7ff3d000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0004220 con 0x7ff3dc1029d0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3d00043c0 con 0x7ff3dc1029d0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3dc1a0970 con 0x7ff3dc1029d0 2024-04-25T18:35:22.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d0003ce0 con 0x7ff3dc1029d0 2024-04-25T18:35:22.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.952+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3dc1a0f00 con 0x7ff3dc1029d0 2024-04-25T18:35:22.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.953+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3dc069730 con 0x7ff3dc1029d0 2024-04-25T18:35:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.953+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ff3d0018430 con 0x7ff3dc1029d0 2024-04-25T18:35:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.954+0000 7ff3da000700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 0x7ff3c403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.954+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff3d004cca0 con 0x7ff3dc1029d0 2024-04-25T18:35:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.954+0000 7ff3e1000700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 0x7ff3c403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:22.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.955+0000 7ff3e1000700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 0x7ff3c403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009c60 tx=0x7ff3c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:22.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:22.959+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3d001fe20 con 0x7ff3dc1029d0 2024-04-25T18:35:23.169 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:22.869+0000 7fe04613a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:35:23.203 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:22.947+0000 7f43674bf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:35:23.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:23.209+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff3dc062e80 con 0x7ff3c4039890 2024-04-25T18:35:23.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:23.167+0000 7fe04613a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: Reconfiguring daemon mgr.x on smithi063 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: Reconfiguring daemon mgr.x on smithi063 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:23.650 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:23.534+0000 7f43674bf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: Reconfiguring daemon mgr.x on smithi063 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:23.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:23.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:24.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:23.768+0000 7fe04613a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:35:24.066 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:35:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:35:23.982+0000 7fe04613a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:35:24.203 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:23.943+0000 7f43674bf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:35:24.203 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:24.035+0000 7f43674bf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:35:24.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:24.425+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7ff3d0017cd0 con 0x7ff3dc1029d0 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: Standby manager daemon z started 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:35:24.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:24 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:24.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:24.230+0000 7f43674bf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:35:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:35:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: Standby manager daemon z started 2024-04-25T18:35:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:35:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: Standby manager daemon z started 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:35:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:24 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/1572892527' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:25.073 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:24.742+0000 7f43674bf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:35:25.073 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:24.832+0000 7f43674bf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:35:25.410 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:25.072+0000 7f43674bf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:35:25.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:25.203+0000 7f43674bf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:35:25.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:25.296+0000 7f43674bf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:35:25.411 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:25.409+0000 7f43674bf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:35:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-25T18:35:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:35:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-25T18:35:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:35:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-25T18:35:25.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:35:25.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:26.203 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:25.734+0000 7f43674bf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:35:26.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:26.326+0000 7f43674bf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:35:26.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:35:26.538+0000 7f43674bf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:35:26.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:26.795+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mon.2 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7ff3d001d260 con 0x7ff3dc1029d0 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: Standby manager daemon y started 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:35:26.901 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:26 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: Standby manager daemon y started 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:35:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:26 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: Standby manager daemon y started 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:35:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:26 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1186816707' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:35:28.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:27 smithi114 ceph-mon[32988]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-25T18:35:28.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:35:28.126 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:27 smithi063 ceph-mon[28006]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-25T18:35:28.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:35:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:27 smithi070 ceph-mon[32820]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-25T18:35:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:35:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:28 smithi114 ceph-mon[32988]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:28 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]: dispatch 2024-04-25T18:35:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:28 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]': finished 2024-04-25T18:35:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:28 smithi114 ceph-mon[32988]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T18:35:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:28 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:29.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:28 smithi063 ceph-mon[28006]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:29.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:28 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]: dispatch 2024-04-25T18:35:29.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:28 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]': finished 2024-04-25T18:35:29.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:28 smithi063 ceph-mon[28006]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T18:35:29.103 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:28 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:28 smithi070 ceph-mon[32820]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:28 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]: dispatch 2024-04-25T18:35:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:28 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2342599373' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0615248c-d2e1-48aa-ab67-0733aaa339cc"}]': finished 2024-04-25T18:35:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:28 smithi070 ceph-mon[32820]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T18:35:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:28 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:30.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:29 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1173009561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:35:30.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:29 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1173009561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:35:30.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:29 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1173009561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:35:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:30 smithi063 ceph-mon[28006]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:31.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:30 smithi070 ceph-mon[32820]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:31.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:30 smithi114 ceph-mon[32988]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:33.098 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:32 smithi063 ceph-mon[28006]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:33.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:32 smithi070 ceph-mon[32820]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:33.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:32 smithi114 ceph-mon[32988]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:35.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:34 smithi063 ceph-mon[28006]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:34 smithi070 ceph-mon[32820]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:35.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:34 smithi114 ceph-mon[32988]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:36.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:36 smithi063 ceph-mon[28006]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:37.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:36 smithi114 ceph-mon[32988]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:36 smithi070 ceph-mon[32820]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:39.021 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:38 smithi063 ceph-mon[28006]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:38 smithi070 ceph-mon[32820]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:39.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:38 smithi114 ceph-mon[32988]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:39.989 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:39 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T18:35:39.990 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:39 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:40.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:39 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T18:35:40.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:39 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:40.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:39 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T18:35:40.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:39 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:40.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:40 smithi063 ceph-mon[28006]: Deploying daemon osd.0 on smithi063 2024-04-25T18:35:40.998 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:40 smithi063 ceph-mon[28006]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:41.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:40 smithi070 ceph-mon[32820]: Deploying daemon osd.0 on smithi063 2024-04-25T18:35:41.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:40 smithi070 ceph-mon[32820]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:41.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:40 smithi114 ceph-mon[32988]: Deploying daemon osd.0 on smithi063 2024-04-25T18:35:41.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:40 smithi114 ceph-mon[32988]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:43.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:42 smithi063 ceph-mon[28006]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:43.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:42 smithi070 ceph-mon[32820]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:43.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:42 smithi114 ceph-mon[32988]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:44 smithi063 ceph-mon[28006]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:44 smithi070 ceph-mon[32820]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:45.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:44 smithi114 ceph-mon[32988]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:47.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:46 smithi063 ceph-mon[28006]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:46 smithi070 ceph-mon[32820]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:47.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:46 smithi114 ceph-mon[32988]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:49.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:48 smithi063 ceph-mon[28006]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:49.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:49.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:48 smithi114 ceph-mon[32988]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:49.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:48 smithi070 ceph-mon[32820]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:50.985 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:50 smithi063 ceph-mon[28006]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:51.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:50 smithi114 ceph-mon[32988]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:50 smithi070 ceph-mon[32820]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:52.148 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.148 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.148 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:52.148 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:52.148 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.149 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:51 smithi063 ceph-mon[28006]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:51 smithi070 ceph-mon[32820]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T18:35:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:35:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:51 smithi114 ceph-mon[32988]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T18:35:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:52 smithi070 ceph-mon[32820]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:52 smithi070 ceph-mon[32820]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T18:35:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:52 smithi070 ceph-mon[32820]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T18:35:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:52 smithi070 ceph-mon[32820]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:35:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:52 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:53.276 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 0 on host 'smithi063' 2024-04-25T18:35:53.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.274+0000 7ff3da000700 1 -- 172.21.15.63:0/2416359482 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff3dc062e80 con 0x7ff3c4039890 2024-04-25T18:35:53.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:52 smithi063 ceph-mon[28006]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:53.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:52 smithi063 ceph-mon[28006]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T18:35:53.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:52 smithi063 ceph-mon[28006]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T18:35:53.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:52 smithi063 ceph-mon[28006]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:35:53.277 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:52 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:53.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 msgr2=0x7ff3c403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:53.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 0x7ff3c403bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009c60 tx=0x7ff3c8009260 comp rx=0 tx=0).stop 2024-04-25T18:35:53.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 msgr2=0x7ff3dc198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff3d000e970 tx=0x7ff3d000ec80 comp rx=0 tx=0).stop 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 shutdown_connections 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff3c4039890 0x7ff3c403bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff3dc104570 0x7ff3dc19ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3dc1029d0 0x7ff3dc198a20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 --2- 172.21.15.63:0/2416359482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3dc103bd0 0x7ff3dc198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.278+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 >> 172.21.15.63:0/2416359482 conn(0x7ff3dc0fdf70 msgr2=0x7ff3dc106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.279+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 shutdown_connections 2024-04-25T18:35:53.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:35:53.279+0000 7ff3e2e00700 1 -- 172.21.15.63:0/2416359482 wait complete. 2024-04-25T18:35:53.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:52 smithi114 ceph-mon[32988]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:53.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:52 smithi114 ceph-mon[32988]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T18:35:53.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:52 smithi114 ceph-mon[32988]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T18:35:53.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:52 smithi114 ceph-mon[32988]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:35:53.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:52 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:53.999 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:54.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:35:54.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T18:35:54.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:54.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:54.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:54.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:54.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:54.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:54.218 DEBUG:teuthology.orchestra.run.smithi063:osd.0> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T18:35:54.224 INFO:tasks.cephadm:Deploying osd.1 on smithi070 with /dev/vg_nvme/lv_4... 2024-04-25T18:35:54.224 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:35:54.466 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:35:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:54 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:35:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:54 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:35:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:54 smithi063 ceph-mon[28006]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' 2024-04-25T18:35:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:54 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:35:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:54 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:35:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:54 smithi070 ceph-mon[32820]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' 2024-04-25T18:35:55.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:54 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:35:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:54 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:35:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:54 smithi114 ceph-mon[32988]: from='osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214]' entity='osd.0' 2024-04-25T18:35:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:55 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:55 smithi063 ceph-mon[28006]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:55 smithi063 ceph-mon[28006]: osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] boot 2024-04-25T18:35:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:55 smithi063 ceph-mon[28006]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T18:35:56.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:55 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00843739 s, 1.2 GB/s 2024-04-25T18:35:56.192 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-25T18:35:56.192 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:55 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:55 smithi070 ceph-mon[32820]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:56.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:55 smithi070 ceph-mon[32820]: osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] boot 2024-04-25T18:35:56.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:55 smithi070 ceph-mon[32820]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T18:35:56.193 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:55 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:55 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:55 smithi114 ceph-mon[32988]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T18:35:56.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:55 smithi114 ceph-mon[32988]: osd.0 [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] boot 2024-04-25T18:35:56.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:55 smithi114 ceph-mon[32988]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T18:35:56.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:55 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:35:56.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-04-25T18:35:57.043 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:35:57.271 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:57 smithi070 ceph-mon[32820]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T18:35:57.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:57 smithi114 ceph-mon[32988]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T18:35:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:57 smithi063 ceph-mon[28006]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T18:35:58.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:58 smithi070 ceph-mon[32820]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:35:58.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:58 smithi114 ceph-mon[32988]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:35:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:58 smithi063 ceph-mon[28006]: pgmap v61: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:35:58.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.502+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/3280855982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18104790 msgr2=0x7f6c1810b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:58.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.502+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/3280855982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18104790 0x7f6c1810b040 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000b0a0 tx=0x7f6c1000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/3280855982 shutdown_connections 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/3280855982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18100d70 0x7f6c18101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/3280855982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18104790 0x7f6c1810b040 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/3280855982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c18103df0 0x7f6c18104250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/3280855982 >> 172.21.15.70:0/3280855982 conn(0x7f6c180fc6b0 msgr2=0x7f6c180feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/3280855982 shutdown_connections 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/3280855982 wait complete. 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 Processor -- start 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- start start 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18103df0 0x7f6c1819b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c18104790 0x7f6c181a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810e630 con 0x7f6c18104790 2024-04-25T18:35:58.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810e4b0 con 0x7f6c18100d70 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.503+0000 7f6c1f5d2700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1810e7b0 con 0x7f6c18103df0 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:60822/0 (socket says 172.21.15.70:60822) 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 -- 172.21.15.70:0/2336024487 learned_addr learned my addr 172.21.15.70:0/2336024487 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1cb6d700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18103df0 0x7f6c1819b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 -- 172.21.15.70:0/2336024487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18103df0 msgr2=0x7f6c1819b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18103df0 0x7f6c1819b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 -- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c18104790 msgr2=0x7f6c181a24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c18104790 0x7f6c181a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 -- 172.21.15.70:0/2336024487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c181a2b40 con 0x7f6c18100d70 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1d36e700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400b500 tx=0x7f6c1400b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14015020 con 0x7f6c18100d70 2024-04-25T18:35:58.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c14011ce0 con 0x7f6c18100d70 2024-04-25T18:35:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c181a2e30 con 0x7f6c18100d70 2024-04-25T18:35:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14010d50 con 0x7f6c18100d70 2024-04-25T18:35:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.504+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c181a3300 con 0x7f6c18100d70 2024-04-25T18:35:58.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.505+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c181a35c0 con 0x7f6c18100d70 2024-04-25T18:35:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.506+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f6c14018020 con 0x7f6c18100d70 2024-04-25T18:35:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.507+0000 7f6c0e7fc700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 0x7f6c040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:35:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.507+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f6c140ce100 con 0x7f6c18100d70 2024-04-25T18:35:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.507+0000 7f6c1cb6d700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 0x7f6c040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:35:58.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.508+0000 7f6c1cb6d700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 0x7f6c040a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:35:58.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.512+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c1401f020 con 0x7f6c18100d70 2024-04-25T18:35:58.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:35:58.719+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6c181a3790 con 0x7f6c040a3bb0 2024-04-25T18:35:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:59 smithi114 ceph-mon[32988]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:35:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:59.336 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:59 smithi070 ceph-mon[32820]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:59.336 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:59.336 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:59.336 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:35:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:35:59.347 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:59 smithi063 ceph-mon[28006]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:35:59.347 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:35:59.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:35:59.348 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:35:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi063 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 17089M 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi063 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 17089M 2024-04-25T18:36:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi063 2024-04-25T18:36:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 17089M 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:01.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:01 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/4292172041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:01 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:01 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]': finished 2024-04-25T18:36:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:01 smithi114 ceph-mon[32988]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T18:36:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:01 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:01 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/4292172041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:01 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:01 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]': finished 2024-04-25T18:36:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:01 smithi063 ceph-mon[28006]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T18:36:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:01 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:01 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/4292172041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:01 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]: dispatch 2024-04-25T18:36:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:01 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c5dd2fb-256f-461a-aa35-959c06ed8333"}]': finished 2024-04-25T18:36:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:01 smithi070 ceph-mon[32820]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T18:36:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:01 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:02 smithi114 ceph-mon[32988]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:02 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/1447387026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:02 smithi063 ceph-mon[28006]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:02 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/1447387026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:02 smithi070 ceph-mon[32820]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:02 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/1447387026' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:04.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:04 smithi114 ceph-mon[32988]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:04 smithi063 ceph-mon[28006]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:04.944 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:04 smithi070 ceph-mon[32820]: pgmap v65: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:06.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:06 smithi114 ceph-mon[32988]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:06 smithi063 ceph-mon[28006]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:06.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:06 smithi070 ceph-mon[32820]: pgmap v66: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:08.624 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:08 smithi070 ceph-mon[32820]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:08 smithi114 ceph-mon[32988]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:08 smithi063 ceph-mon[28006]: pgmap v67: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:09.606 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T18:36:09.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:09.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:09 smithi070 ceph-mon[32820]: Deploying daemon osd.1 on smithi070 2024-04-25T18:36:09.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T18:36:09.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:09.904 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:09 smithi063 ceph-mon[28006]: Deploying daemon osd.1 on smithi070 2024-04-25T18:36:09.904 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T18:36:09.905 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:09.905 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:09 smithi114 ceph-mon[32988]: Deploying daemon osd.1 on smithi070 2024-04-25T18:36:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:10 smithi114 ceph-mon[32988]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:10 smithi063 ceph-mon[28006]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:10.905 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:10 smithi070 ceph-mon[32820]: pgmap v68: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:12 smithi070 ceph-mon[32820]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:12.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:12 smithi114 ceph-mon[32988]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:12 smithi063 ceph-mon[28006]: pgmap v69: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:14 smithi114 ceph-mon[32988]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:14.874 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:14 smithi070 ceph-mon[32820]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:14.874 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.874 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.874 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:14 smithi063 ceph-mon[28006]: pgmap v70: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:16 smithi114 ceph-mon[32988]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:16.839 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:16 smithi070 ceph-mon[32820]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:16.876 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:16 smithi063 ceph-mon[28006]: pgmap v71: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:17.962 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 1 on host 'smithi070' 2024-04-25T18:36:17.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.960+0000 7f6c0e7fc700 1 -- 172.21.15.70:0/2336024487 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c181a3790 con 0x7f6c040a3bb0 2024-04-25T18:36:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 msgr2=0x7f6c040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 0x7f6c040a6070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6c08006fd0 tx=0x7f6c08008040 comp rx=0 tx=0).stop 2024-04-25T18:36:17.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 msgr2=0x7f6c1819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400b500 tx=0x7f6c1400b8c0 comp rx=0 tx=0).stop 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 shutdown_connections 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c040a3bb0 0x7f6c040a6070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c18104790 0x7f6c181a24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c18100d70 0x7f6c1819aee0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 --2- 172.21.15.70:0/2336024487 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c18103df0 0x7f6c1819b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 >> 172.21.15.70:0/2336024487 conn(0x7f6c180fc6b0 msgr2=0x7f6c18101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 shutdown_connections 2024-04-25T18:36:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:36:17.962+0000 7f6c1f5d2700 1 -- 172.21.15.70:0/2336024487 wait complete. 2024-04-25T18:36:18.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:18.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:18.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:18.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:18.125 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:18.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.585 DEBUG:teuthology.orchestra.run.smithi070:osd.1> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T18:36:18.587 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_4... 2024-04-25T18:36:18.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T18:36:18.831 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:36:18.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:18.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:18.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:18.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:18.953 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:18 smithi114 ceph-mon[32988]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:19.016 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:19.016 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:19.016 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:19.016 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:19.017 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:19.017 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:18 smithi070 ceph-mon[32820]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: pgmap v72: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:19.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:18 smithi063 ceph-mon[28006]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T18:36:20.203 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1[37434]: 2024-04-25T18:36:19.958+0000 7f456abb6700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T18:36:20.203 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1[37434]: 2024-04-25T18:36:19.961+0000 7f45669a9700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:19 smithi070 ceph-mon[32820]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.283 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:19 smithi114 ceph-mon[32988]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: from='osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:36:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:19 smithi063 ceph-mon[28006]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112914 s, 929 MB/s 2024-04-25T18:36:20.870 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-25T18:36:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:20 smithi063 ceph-mon[28006]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:36:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:20 smithi063 ceph-mon[28006]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T18:36:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:20 smithi114 ceph-mon[32988]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:36:21.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:20 smithi114 ceph-mon[32988]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T18:36:21.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.194 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.317 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:20 smithi070 ceph-mon[32820]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:36:21.318 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:20 smithi070 ceph-mon[32820]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T18:36:21.318 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.318 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:21.555 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2024-04-25T18:36:21.790 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:36:22.205 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] boot 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi070 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:22.206 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:22.207 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:36:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] boot 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi070 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: osd.1 [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] boot 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: pgmap v77: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi070 2024-04-25T18:36:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 17088M 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:22.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:23.098 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:22 smithi114 ceph-mon[32988]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T18:36:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 -- 172.21.15.114:0/4105458664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 msgr2=0x7f7650105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 --2- 172.21.15.114:0/4105458664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650105920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7640009990 tx=0x7f7640009ca0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.332 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 -- 172.21.15.114:0/4105458664 shutdown_connections 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 --2- 172.21.15.114:0/4105458664 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650105920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 --2- 172.21.15.114:0/4105458664 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 0x7f765010ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 --2- 172.21.15.114:0/4105458664 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7650105e60 0x7f7650108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.330+0000 7f7654fdf700 1 -- 172.21.15.114:0/4105458664 >> 172.21.15.114:0/4105458664 conn(0x7f76500fcf60 msgr2=0x7f76500ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.331+0000 7f7654fdf700 1 -- 172.21.15.114:0/4105458664 shutdown_connections 2024-04-25T18:36:23.333 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.331+0000 7f7654fdf700 1 -- 172.21.15.114:0/4105458664 wait complete. 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.331+0000 7f7654fdf700 1 Processor -- start 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.331+0000 7f7654fdf700 1 -- start start 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7650105e60 0x7f76501991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 0x7f76501a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f765010e620 con 0x7f7650105e60 2024-04-25T18:36:23.335 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f765010e4a0 con 0x7f7650108790 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f765010e7a0 con 0x7f7650103530 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:34852/0 (socket says 172.21.15.114:34852) 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 -- 172.21.15.114:0/603171050 learned_addr learned my addr 172.21.15.114:0/603171050 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764ed9d700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 0x7f76501a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 -- 172.21.15.114:0/603171050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 msgr2=0x7f76501a0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 0x7f76501a0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 -- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7650105e60 msgr2=0x7f76501991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7650105e60 0x7f76501991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 -- 172.21.15.114:0/603171050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76501a0920 con 0x7f7650103530 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f764e59c700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76400036f0 tx=0x7f76400057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640026070 con 0x7f7650103530 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7640004080 con 0x7f7650103530 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7650101320 con 0x7f7650103530 2024-04-25T18:36:23.336 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.332+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764000fe20 con 0x7f7650103530 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.333+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76501018d0 con 0x7f7650103530 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.333+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7650069730 con 0x7f7650103530 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.335+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f7640029070 con 0x7f7650103530 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.335+0000 7f763f7fe700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 0x7f76380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.335+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f76400cfb80 con 0x7f7650103530 2024-04-25T18:36:23.337 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.336+0000 7f764dd9b700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 0x7f76380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:23.338 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.337+0000 7f764dd9b700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 0x7f76380a60f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f765019a1b0 tx=0x7f7644006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:36:23.340 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.339+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7640030080 con 0x7f7650103530 2024-04-25T18:36:23.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:22 smithi063 ceph-mon[28006]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T18:36:23.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:22 smithi070 ceph-mon[32820]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T18:36:23.544 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:23.542+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7650062e80 con 0x7f76380a3c30 2024-04-25T18:36:24.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:23 smithi114 ceph-mon[32988]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:24.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:23 smithi114 ceph-mon[32988]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:24.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:24.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:24.282 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:23 smithi063 ceph-mon[28006]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:23 smithi063 ceph-mon[28006]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:23 smithi070 ceph-mon[32820]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:23 smithi070 ceph-mon[32820]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:24.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:24.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:26.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:26 smithi114 ceph-mon[32988]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:26.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:26 smithi063 ceph-mon[28006]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:26.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:26 smithi070 ceph-mon[32820]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/82003317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]': finished 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:27.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:27 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/309716824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/82003317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]': finished 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:27.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:27 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/309716824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/82003317' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]: dispatch 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "719d6679-f15c-4e02-b8c9-e19a26526679"}]': finished 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:27.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:27 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/309716824' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:28.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:28 smithi114 ceph-mon[32988]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:28 smithi063 ceph-mon[28006]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:28 smithi070 ceph-mon[32820]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:30.509 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:30 smithi114 ceph-mon[32988]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:30 smithi063 ceph-mon[28006]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:30 smithi070 ceph-mon[32820]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:32.549 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:32 smithi114 ceph-mon[32988]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:32 smithi063 ceph-mon[28006]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:32 smithi070 ceph-mon[32820]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:34 smithi114 ceph-mon[32988]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:34 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T18:36:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:34 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:34 smithi114 ceph-mon[32988]: Deploying daemon osd.2 on smithi114 2024-04-25T18:36:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:34 smithi063 ceph-mon[28006]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:34 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T18:36:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:34 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:34 smithi063 ceph-mon[28006]: Deploying daemon osd.2 on smithi114 2024-04-25T18:36:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:34 smithi070 ceph-mon[32820]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:34 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T18:36:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:34 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:34 smithi070 ceph-mon[32820]: Deploying daemon osd.2 on smithi114 2024-04-25T18:36:36.459 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:36 smithi114 ceph-mon[32988]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:36 smithi063 ceph-mon[28006]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:36 smithi070 ceph-mon[32820]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:38.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:38 smithi114 ceph-mon[32988]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:38 smithi063 ceph-mon[28006]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:38 smithi070 ceph-mon[32820]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:40 smithi063 ceph-mon[28006]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:40 smithi070 ceph-mon[32820]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:40 smithi114 ceph-mon[32988]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:42.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:42 smithi063 ceph-mon[28006]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:42 smithi070 ceph-mon[32820]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:42.460 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:42 smithi114 ceph-mon[32988]: pgmap v89: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:42.532 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2024-04-25T18:36:42.533 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.531+0000 7f763f7fe700 1 -- 172.21.15.114:0/603171050 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7650062e80 con 0x7f76380a3c30 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 msgr2=0x7f76380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 0x7f76380a60f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f765019a1b0 tx=0x7f7644006d20 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 msgr2=0x7f7650198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f76400036f0 tx=0x7f76400057c0 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 shutdown_connections 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f76380a3c30 0x7f76380a60f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7650103530 0x7f7650198ca0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7650108790 0x7f76501a0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 --2- 172.21.15.114:0/603171050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7650105e60 0x7f76501991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:42.535 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 >> 172.21.15.114:0/603171050 conn(0x7f76500fcf60 msgr2=0x7f7650103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:36:42.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 shutdown_connections 2024-04-25T18:36:42.536 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:36:42.533+0000 7f7654fdf700 1 -- 172.21.15.114:0/603171050 wait complete. 2024-04-25T18:36:43.160 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T18:36:43.162 INFO:tasks.cephadm:Deploying osd.3 on smithi063 with /dev/vg_nvme/lv_3... 2024-04-25T18:36:43.162 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T18:36:43.598 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:43.755 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:43 smithi063 ceph-mon[28006]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:43.784 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:43 smithi114 ceph-mon[32988]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:43 smithi070 ceph-mon[32820]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:45.035 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:44 smithi063 ceph-mon[28006]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-mon[32988]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:45.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2[38142]: 2024-04-25T18:36:44.829+0000 7f95a85c2700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T18:36:45.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2[38142]: 2024-04-25T18:36:44.833+0000 7f95a43b5700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: from='osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:44 smithi070 ceph-mon[32820]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:36:46.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:45 smithi114 ceph-mon[32988]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:36:46.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:45 smithi114 ceph-mon[32988]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T18:36:46.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:45 smithi063 ceph-mon[28006]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:36:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:45 smithi063 ceph-mon[28006]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T18:36:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:45 smithi070 ceph-mon[32820]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:36:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:45 smithi070 ceph-mon[32820]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T18:36:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:46.633 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T18:36:46.633 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T18:36:46.633 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T18:36:46.633 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T18:36:46.634 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0378425 s, 277 MB/s 2024-04-25T18:36:46.634 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 2024-04-25T18:36:46.634 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T18:36:47.086 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] boot 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.087 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:47.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.088 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] boot 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.106 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:47.106 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.106 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.106 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: osd.2 [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] boot 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:36:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:36:47.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:47.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-04-25T18:36:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:36:48.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:47 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi114 2024-04-25T18:36:48.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:47 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-25T18:36:48.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:47 smithi063 ceph-mon[28006]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T18:36:48.173 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:48.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:47 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi114 2024-04-25T18:36:48.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:47 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-25T18:36:48.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:47 smithi070 ceph-mon[32820]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T18:36:48.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:48.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:47 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi114 2024-04-25T18:36:48.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:47 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 17088M 2024-04-25T18:36:48.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:47 smithi114 ceph-mon[32988]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T18:36:48.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:49.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:48 smithi063 ceph-mon[28006]: pgmap v96: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:49.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:49.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:48 smithi063 ceph-mon[28006]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T18:36:49.143 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:49.143 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.143 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.144 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47421]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.144 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47421]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:48 smithi070 ceph-mon[32820]: pgmap v96: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:48 smithi070 ceph-mon[32820]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T18:36:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:49.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:48 smithi114 ceph-mon[32988]: pgmap v96: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:49.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:49.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:48 smithi114 ceph-mon[32988]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T18:36:49.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T18:36:49.440 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.440 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.440 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47424]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.440 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47424]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41267]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41267]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41267]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41270]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41270]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41270]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.566 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41270]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41903]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41903]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41903]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41903]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.758 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41906]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41897]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41897]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41897]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41897]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41900]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41900]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41900]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.759 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 18:36:49 smithi114 sudo[41900]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47427]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47427]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47430]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47430]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47430]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.805 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 sudo[47430]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.913 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T18:36:49.913 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41274]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41274]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41277]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T18:36:49.914 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 sudo[41277]: pam_unix(sudo:session): session closed for user root 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:49 smithi114 ceph-mon[32988]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:49 smithi063 ceph-mon[28006]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T18:36:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T18:36:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:36:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:36:50.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:49 smithi070 ceph-mon[32820]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T18:36:50.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 -- 172.21.15.63:0/481976809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 msgr2=0x7f75a0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 --2- 172.21.15.63:0/481976809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a0102f10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7590009990 tx=0x7f7590009ca0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 -- 172.21.15.63:0/481976809 shutdown_connections 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 --2- 172.21.15.63:0/481976809 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a0102f10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 --2- 172.21.15.63:0/481976809 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 0x7f75a010af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 --2- 172.21.15.63:0/481976809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 0x7f75a0104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 -- 172.21.15.63:0/481976809 >> 172.21.15.63:0/481976809 conn(0x7f75a00fe090 msgr2=0x7f75a01004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 -- 172.21.15.63:0/481976809 shutdown_connections 2024-04-25T18:36:50.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.486+0000 7f75a4e00700 1 -- 172.21.15.63:0/481976809 wait complete. 2024-04-25T18:36:50.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.488+0000 7f75a4e00700 1 Processor -- start 2024-04-25T18:36:50.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.488+0000 7f75a4e00700 1 -- start start 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 0x7f75a019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 0x7f75a01a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010e5d0 con 0x7f75a0103cf0 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010e450 con 0x7f75a0104690 2024-04-25T18:36:50.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f75a4e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75a010e750 con 0x7f75a0102af0 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f759d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 0x7f75a019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f759de00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f759de00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45336/0 (socket says 172.21.15.63:45336) 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f759de00700 1 -- 172.21.15.63:0/2011401173 learned_addr learned my addr 172.21.15.63:0/2011401173 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.489+0000 7f759e800700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 0x7f75a01a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 -- 172.21.15.63:0/2011401173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 msgr2=0x7f75a01a2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 0x7f75a01a2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 -- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 msgr2=0x7f75a019b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 0x7f75a019b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:36:50.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 -- 172.21.15.63:0/2011401173 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a01a2b30 con 0x7f75a0102af0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.490+0000 7f759de00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7590000c00 tx=0x7f7590005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.491+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590027070 con 0x7f75a0102af0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.491+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7590022470 con 0x7f75a0102af0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.491+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a01a2dc0 con 0x7f75a0102af0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.491+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a01a31c0 con 0x7f75a0102af0 2024-04-25T18:36:50.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.491+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7590017630 con 0x7f75a0102af0 2024-04-25T18:36:50.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.492+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a01a33f0 con 0x7f75a0102af0 2024-04-25T18:36:50.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.494+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f759002a070 con 0x7f75a0102af0 2024-04-25T18:36:50.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.497+0000 7f7596a00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 0x7f75840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:36:50.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.497+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3237+0+0 (secure 0 0 0) 0x7f75900cfcf0 con 0x7f75a0102af0 2024-04-25T18:36:50.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.497+0000 7f759d400700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 0x7f75840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:36:50.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.498+0000 7f759d400700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 0x7f75840a6180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f758c009c60 tx=0x7f758c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:36:50.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.499+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7590013070 con 0x7f75a0102af0 2024-04-25T18:36:50.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:36:50.762+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f75a0076390 con 0x7f75840a3cc0 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: pgmap v99: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: pgmap v99: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:51.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: pgmap v99: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:36:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:36:52.183 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:51 smithi063 ceph-mon[28006]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:52.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:51 smithi070 ceph-mon[32820]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:51 smithi114 ceph-mon[32988]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:54.111 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:54 smithi063 ceph-mon[28006]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:54.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:54 smithi070 ceph-mon[32820]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:54.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:54 smithi114 ceph-mon[32988]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:56.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:56.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3825012571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]': finished 2024-04-25T18:36:56.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T18:36:56.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3825012571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]': finished 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T18:36:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3825012571' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]: dispatch 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5052332b-1c7e-42e7-9ae6-4751b86b2986"}]': finished 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T18:36:56.570 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:36:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:57 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4287638417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:57.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:57 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4287638417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:57.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:57 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4287638417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:36:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:36:58 smithi063 ceph-mon[28006]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:58.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:36:58 smithi070 ceph-mon[32820]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:36:58.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:36:58 smithi114 ceph-mon[32988]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:00.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:00 smithi063 ceph-mon[28006]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:00.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:00 smithi070 ceph-mon[32820]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:00.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:00 smithi114 ceph-mon[32988]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:02.269 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:02 smithi063 ceph-mon[28006]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:02.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:02 smithi070 ceph-mon[32820]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:02.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:02 smithi114 ceph-mon[32988]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:04.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:04 smithi063 ceph-mon[28006]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:04.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:04 smithi070 ceph-mon[32820]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:04.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:04 smithi114 ceph-mon[32988]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:06 smithi063 ceph-mon[28006]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T18:37:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:06 smithi063 ceph-mon[28006]: Deploying daemon osd.3 on smithi063 2024-04-25T18:37:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:06 smithi070 ceph-mon[32820]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T18:37:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:06 smithi070 ceph-mon[32820]: Deploying daemon osd.3 on smithi063 2024-04-25T18:37:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:06 smithi114 ceph-mon[32988]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T18:37:06.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:06.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:06 smithi114 ceph-mon[32988]: Deploying daemon osd.3 on smithi063 2024-04-25T18:37:08.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:08 smithi063 ceph-mon[28006]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:08.387 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:08 smithi114 ceph-mon[32988]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:08.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:08 smithi070 ceph-mon[32820]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:10 smithi063 ceph-mon[28006]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:10 smithi070 ceph-mon[32820]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:10.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:10 smithi114 ceph-mon[32988]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:12 smithi070 ceph-mon[32820]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:12.510 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:12 smithi063 ceph-mon[28006]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:12.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:12 smithi114 ceph-mon[32988]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:14.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:14 smithi070 ceph-mon[32820]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:14.539 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:14 smithi063 ceph-mon[28006]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:14.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:14 smithi114 ceph-mon[32988]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:15.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:15.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:15.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:16 smithi063 ceph-mon[28006]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:16 smithi070 ceph-mon[32820]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:16 smithi114 ceph-mon[32988]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:18 smithi063 ceph-mon[28006]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:18 smithi070 ceph-mon[32820]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:18 smithi114 ceph-mon[32988]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:19.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:19 smithi070 ceph-mon[32820]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T18:37:19.728 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:19 smithi063 ceph-mon[28006]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T18:37:19.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.725+0000 7f7596a00700 1 -- 172.21.15.63:0/2011401173 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f75a0076390 con 0x7f75840a3cc0 2024-04-25T18:37:19.728 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 3 on host 'smithi063' 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.729+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 msgr2=0x7f75840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.729+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 0x7f75840a6180 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f758c009c60 tx=0x7f758c009260 comp rx=0 tx=0).stop 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 msgr2=0x7f75a019af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7590000c00 tx=0x7f7590005b00 comp rx=0 tx=0).stop 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 shutdown_connections 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f75840a3cc0 0x7f75840a6180 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f75a0102af0 0x7f75a019af20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75a0104690 0x7f75a01a2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 --2- 172.21.15.63:0/2011401173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a0103cf0 0x7f75a019b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 >> 172.21.15.63:0/2011401173 conn(0x7f75a00fe090 msgr2=0x7f75a0074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:37:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 shutdown_connections 2024-04-25T18:37:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:37:19.730+0000 7f75a4e00700 1 -- 172.21.15.63:0/2011401173 wait complete. 2024-04-25T18:37:19.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:19 smithi114 ceph-mon[32988]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T18:37:20.603 DEBUG:teuthology.orchestra.run.smithi063:osd.3> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T18:37:20.608 INFO:tasks.cephadm:Deploying osd.4 on smithi070 with /dev/vg_nvme/lv_3... 2024-04-25T18:37:20.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:20.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:20.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:20.852 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:37:20.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:20.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T18:37:20.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T18:37:20.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:37:20.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:20.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:21.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-mon[28006]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:37:21.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-mon[28006]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T18:37:21.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.492 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-mon[28006]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' 2024-04-25T18:37:21.493 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T18:37:21.423+0000 7f8e32400700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T18:37:21.493 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 18:37:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T18:37:21.489+0000 7f8e28200700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:37:21.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:21 smithi070 ceph-mon[32820]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:37:21.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:21 smithi070 ceph-mon[32820]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T18:37:21.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:21 smithi070 ceph-mon[32820]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' 2024-04-25T18:37:21.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:21 smithi114 ceph-mon[32988]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:37:21.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:21 smithi114 ceph-mon[32988]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T18:37:21.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:21.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:21 smithi114 ceph-mon[32988]: from='osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680]' entity='osd.3' 2024-04-25T18:37:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:22 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:37:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:22 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:37:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:22 smithi063 ceph-mon[28006]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:22.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:22 smithi063 ceph-mon[28006]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T18:37:22.702 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:22 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:37:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:22 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:37:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:22 smithi070 ceph-mon[32820]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:22 smithi070 ceph-mon[32820]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T18:37:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00881051 s, 1.2 GB/s 2024-04-25T18:37:22.711 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-25T18:37:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:22 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:37:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:22 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:37:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:22 smithi114 ceph-mon[32988]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:22 smithi114 ceph-mon[32988]: osdmap e26: 4 total, 3 up, 4 in 2024-04-25T18:37:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:23.307 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-04-25T18:37:23.548 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:37:23.582 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:23 smithi070 ceph-mon[32820]: osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] boot 2024-04-25T18:37:23.582 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:23 smithi070 ceph-mon[32820]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T18:37:23.582 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:23 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:23.582 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:23 smithi070 ceph-mon[32820]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T18:37:23.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:23 smithi114 ceph-mon[32988]: osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] boot 2024-04-25T18:37:23.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:23 smithi114 ceph-mon[32988]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T18:37:23.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:23 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:23.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:23 smithi114 ceph-mon[32988]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T18:37:23.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:23 smithi063 ceph-mon[28006]: osd.3 [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] boot 2024-04-25T18:37:23.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:23 smithi063 ceph-mon[28006]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T18:37:23.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:23 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:37:23.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:23 smithi063 ceph-mon[28006]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T18:37:24.686 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:24 smithi070 ceph-mon[32820]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:24.687 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:24 smithi070 ceph-mon[32820]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T18:37:24.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:24 smithi114 ceph-mon[32988]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:24.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:24 smithi114 ceph-mon[32988]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T18:37:24.837 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:24 smithi063 ceph-mon[28006]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-25T18:37:24.837 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:24 smithi063 ceph-mon[28006]: osdmap e29: 4 total, 4 up, 4 in 2024-04-25T18:37:25.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 -- 172.21.15.70:0/719292259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600104660 msgr2=0x7fc60010af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:25.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 --2- 172.21.15.70:0/719292259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600104660 0x7fc60010af10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800b0d0 tx=0x7fc5f800b3e0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 -- 172.21.15.70:0/719292259 shutdown_connections 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 --2- 172.21.15.70:0/719292259 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 --2- 172.21.15.70:0/719292259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600104660 0x7fc60010af10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 --2- 172.21.15.70:0/719292259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc600103cc0 0x7fc600104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 -- 172.21.15.70:0/719292259 >> 172.21.15.70:0/719292259 conn(0x7fc6000fe060 msgr2=0x7fc6001004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 -- 172.21.15.70:0/719292259 shutdown_connections 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.282+0000 7fc607b51700 1 -- 172.21.15.70:0/719292259 wait complete. 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 Processor -- start 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 -- start start 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600198d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:25.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc600104660 0x7fc6001a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc60010e650 con 0x7fc600104660 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc60010e4d0 con 0x7fc600103cc0 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc607b51700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc60010e7d0 con 0x7fc600102ac0 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:55084/0 (socket says 172.21.15.70:55084) 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 -- 172.21.15.70:0/721686966 learned_addr learned my addr 172.21.15.70:0/721686966 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6058ed700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600198d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 -- 172.21.15.70:0/721686966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 msgr2=0x7fc600198d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600198d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 -- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc600104660 msgr2=0x7fc6001a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc600104660 0x7fc6001a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 -- 172.21.15.70:0/721686966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6001a0950 con 0x7fc600103cc0 2024-04-25T18:37:25.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6058ed700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.283+0000 7fc6050ec700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000b530 tx=0x7fc5f000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.284+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0015020 con 0x7fc600103cc0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.284+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6001a0c40 con 0x7fc600103cc0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.284+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5f0011ce0 con 0x7fc600103cc0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.284+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0010d20 con 0x7fc600103cc0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.284+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6001a1200 con 0x7fc600103cc0 2024-04-25T18:37:25.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.285+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc600069730 con 0x7fc600103cc0 2024-04-25T18:37:25.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.286+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc5f0018020 con 0x7fc600103cc0 2024-04-25T18:37:25.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.286+0000 7fc5f6ffd700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 0x7fc5ec0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:25.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.286+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3698+0+0 (secure 0 0 0) 0x7fc5f00cea40 con 0x7fc600103cc0 2024-04-25T18:37:25.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.287+0000 7fc6058ed700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 0x7fc5ec0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:25.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.288+0000 7fc6058ed700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 0x7fc5ec0a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc60019a1e0 tx=0x7fc5fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:37:25.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.291+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5f001f020 con 0x7fc600103cc0 2024-04-25T18:37:25.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:25.502+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc600062e80 con 0x7fc5ec0a3c40 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:26.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:26.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: osdmap e30: 4 total, 4 up, 4 in 2024-04-25T18:37:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:27.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:27 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi063 2024-04-25T18:37:27.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:27 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T18:37:27.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:27 smithi070 ceph-mon[32820]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T18:37:27.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:27 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi063 2024-04-25T18:37:27.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:27 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T18:37:27.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:27 smithi114 ceph-mon[32988]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T18:37:27.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:27 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi063 2024-04-25T18:37:27.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:27 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T18:37:27.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:27 smithi063 ceph-mon[28006]: osdmap e31: 4 total, 4 up, 4 in 2024-04-25T18:37:28.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:37:28.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/1052374191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]': finished 2024-04-25T18:37:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T18:37:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:28 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/1052374191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]': finished 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T18:37:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:28 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:37:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/1052374191' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]: dispatch 2024-04-25T18:37:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0da5a992-e540-4065-a95a-2fe0eb881da5"}]': finished 2024-04-25T18:37:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: osdmap e32: 5 total, 4 up, 5 in 2024-04-25T18:37:28.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:28 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:29.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:29 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/1409062139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:29.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:29 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/1409062139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:29.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:29 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/1409062139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:30.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:30 smithi114 ceph-mon[32988]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T18:37:30.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:30 smithi063 ceph-mon[28006]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T18:37:30.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:30 smithi070 ceph-mon[32820]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-25T18:37:32.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:32 smithi114 ceph-mon[32988]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:37:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:32 smithi063 ceph-mon[28006]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:37:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:32 smithi070 ceph-mon[32820]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:37:34.584 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:34 smithi070 ceph-mon[32820]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T18:37:34.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:34 smithi114 ceph-mon[32988]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T18:37:34.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:34 smithi063 ceph-mon[28006]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T18:37:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:36 smithi114 ceph-mon[32988]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T18:37:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:36 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T18:37:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:36 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:36 smithi114 ceph-mon[32988]: Deploying daemon osd.4 on smithi070 2024-04-25T18:37:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:36 smithi063 ceph-mon[28006]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T18:37:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T18:37:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:36 smithi063 ceph-mon[28006]: Deploying daemon osd.4 on smithi070 2024-04-25T18:37:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:36 smithi070 ceph-mon[32820]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T18:37:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:36 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T18:37:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:36 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:36 smithi070 ceph-mon[32820]: Deploying daemon osd.4 on smithi070 2024-04-25T18:37:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:38 smithi114 ceph-mon[32988]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:38 smithi063 ceph-mon[28006]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:38.932 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:38 smithi070 ceph-mon[32820]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:40.808 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:40 smithi114 ceph-mon[32988]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:40.857 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:40 smithi070 ceph-mon[32820]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:40 smithi063 ceph-mon[28006]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:42.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:42.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:42.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:42.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:42.993 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:42 smithi070 ceph-mon[32820]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:43.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:42 smithi114 ceph-mon[32988]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:43.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:42 smithi063 ceph-mon[28006]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:43.982 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:43 smithi070 ceph-mon[32820]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 4 on host 'smithi070' 2024-04-25T18:37:44.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.087+0000 7fc5f6ffd700 1 -- 172.21.15.70:0/721686966 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc600062e80 con 0x7fc5ec0a3c40 2024-04-25T18:37:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 msgr2=0x7fc5ec0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 0x7fc5ec0a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc60019a1e0 tx=0x7fc5fc008040 comp rx=0 tx=0).stop 2024-04-25T18:37:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 msgr2=0x7fc600199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000b530 tx=0x7fc5f000b8f0 comp rx=0 tx=0).stop 2024-04-25T18:37:44.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 shutdown_connections 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5ec0a3c40 0x7fc5ec0a6100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc600102ac0 0x7fc600198d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc600104660 0x7fc6001a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 --2- 172.21.15.70:0/721686966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc600103cc0 0x7fc600199240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 >> 172.21.15.70:0/721686966 conn(0x7fc6000fe060 msgr2=0x7fc600107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 shutdown_connections 2024-04-25T18:37:44.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:37:44.090+0000 7fc607b51700 1 -- 172.21.15.70:0/721686966 wait complete. 2024-04-25T18:37:44.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:43 smithi114 ceph-mon[32988]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:44.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:43 smithi063 ceph-mon[28006]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:44.740 DEBUG:teuthology.orchestra.run.smithi070:osd.4> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T18:37:44.742 INFO:tasks.cephadm:Deploying osd.5 on smithi114 with /dev/vg_nvme/lv_3... 2024-04-25T18:37:44.742 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T18:37:44.986 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:45 smithi114 ceph-mon[32988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:45.330 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:45 smithi070 ceph-mon[32820]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:45 smithi063 ceph-mon[28006]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T18:37:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:46.939 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T18:37:46.939 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:46.939 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T18:37:46.939 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:46.939 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-mon[32820]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:46.940 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T18:37:46.426+0000 7f4d0c2a7700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T18:37:46.940 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 18:37:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T18:37:46.433+0000 7f4d0809a700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:46 smithi063 ceph-mon[28006]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: from='osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: osdmap e33: 5 total, 4 up, 5 in 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:46.941 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:46 smithi114 ceph-mon[32988]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:37:47.085 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T18:37:47.085 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T18:37:47.086 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-25T18:37:47.086 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-25T18:37:47.086 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115125 s, 911 MB/s 2024-04-25T18:37:47.086 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-25T18:37:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:47 smithi070 ceph-mon[32820]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:37:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:47 smithi070 ceph-mon[32820]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T18:37:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:47 smithi114 ceph-mon[32988]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:37:47.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:47 smithi114 ceph-mon[32988]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T18:37:47.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.721 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.752 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2024-04-25T18:37:47.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:47 smithi063 ceph-mon[28006]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:37:47.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:47 smithi063 ceph-mon[28006]: osdmap e34: 5 total, 4 up, 5 in 2024-04-25T18:37:47.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:47.994 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:37:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:37:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] boot 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi070 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:48.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] boot 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi070 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: osd.4 [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] boot 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: osdmap e35: 5 total, 5 up, 5 in 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi070 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 8544M 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:37:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:37:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.573+0000 7f84706a3700 1 -- 172.21.15.114:0/2781082613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468104640 msgr2=0x7f846810aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.573+0000 7f84706a3700 1 --2- 172.21.15.114:0/2781082613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468104640 0x7f846810aef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f846000b0a0 tx=0x7f846000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- 172.21.15.114:0/2781082613 shutdown_connections 2024-04-25T18:37:49.575 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- 172.21.15.114:0/2781082613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468104640 0x7f846810aef0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- 172.21.15.114:0/2781082613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468102aa0 0x7f8468102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- 172.21.15.114:0/2781082613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8468103ca0 0x7f8468104100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- 172.21.15.114:0/2781082613 >> 172.21.15.114:0/2781082613 conn(0x7f84680fe040 msgr2=0x7f8468100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:37:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- 172.21.15.114:0/2781082613 shutdown_connections 2024-04-25T18:37:49.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- 172.21.15.114:0/2781082613 wait complete. 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 Processor -- start 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- start start 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8468102aa0 0x7f8468198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 0x7f84681a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f846810e5b0 con 0x7f8468102aa0 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f846810e430 con 0x7f8468104640 2024-04-25T18:37:49.578 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f84706a3700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f846810e730 con 0x7f8468103ca0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.574+0000 7f846dc3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:55076/0 (socket says 172.21.15.114:55076) 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 -- 172.21.15.114:0/1149357643 learned_addr learned my addr 172.21.15.114:0/1149357643 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846ec40700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 0x7f84681a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 -- 172.21.15.114:0/1149357643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 msgr2=0x7f84681a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 0x7f84681a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 -- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8468102aa0 msgr2=0x7f8468198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8468102aa0 0x7f8468198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 -- 172.21.15.114:0/1149357643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84681a0890 con 0x7f8468103ca0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846ec40700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 0x7f84681a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f846dc3e700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f846400b530 tx=0x7f846400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464015020 con 0x7f8468103ca0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8464011ce0 con 0x7f8468103ca0 2024-04-25T18:37:49.579 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84681a0b80 con 0x7f8468103ca0 2024-04-25T18:37:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84681a1110 con 0x7f8468103ca0 2024-04-25T18:37:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.575+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8464010e40 con 0x7f8468103ca0 2024-04-25T18:37:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.576+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8468069730 con 0x7f8468103ca0 2024-04-25T18:37:49.580 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.577+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8464018020 con 0x7f8468103ca0 2024-04-25T18:37:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.578+0000 7f845f7fe700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 0x7f84540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:37:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.578+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4101+0+0 (secure 0 0 0) 0x7f84640ceb90 con 0x7f8468103ca0 2024-04-25T18:37:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.578+0000 7f846e43f700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 0x7f84540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:37:49.581 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.579+0000 7f846e43f700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 0x7f84540a6180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f846819a120 tx=0x7f84580096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:37:49.584 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.583+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f846401f020 con 0x7f8468103ca0 2024-04-25T18:37:49.699 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:49 smithi070 ceph-mon[32820]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T18:37:49.779 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:49 smithi063 ceph-mon[28006]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T18:37:49.790 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:37:49.789+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8468062e80 con 0x7f84540a3cc0 2024-04-25T18:37:49.791 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:49 smithi114 ceph-mon[32988]: osdmap e36: 5 total, 5 up, 5 in 2024-04-25T18:37:50.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:50 smithi114 ceph-mon[32988]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:50.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:50 smithi114 ceph-mon[32988]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:50.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:50.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:50.642 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:50.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:50 smithi063 ceph-mon[28006]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:50.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:50 smithi063 ceph-mon[28006]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:50.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:50.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:50.807 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:50 smithi070 ceph-mon[32820]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:50 smithi070 ceph-mon[32820]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:37:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:37:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:37:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:37:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:52 smithi114 ceph-mon[32988]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:52 smithi063 ceph-mon[28006]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:52.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:52 smithi070 ceph-mon[32820]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/722514797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]': finished 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:37:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:53 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/2762418324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:53.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/722514797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]': finished 2024-04-25T18:37:53.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T18:37:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:37:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:53 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/2762418324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/722514797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]: dispatch 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8832dbef-9e3e-4abf-8ac4-d279ca3af434"}]': finished 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: osdmap e37: 6 total, 5 up, 6 in 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:37:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:53 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/2762418324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:37:54.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:54 smithi114 ceph-mon[32988]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:54 smithi063 ceph-mon[28006]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:54 smithi070 ceph-mon[32820]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:56 smithi114 ceph-mon[32988]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:56 smithi063 ceph-mon[28006]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:56.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:56 smithi070 ceph-mon[32820]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:58.827 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:37:58 smithi114 ceph-mon[32988]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:37:58 smithi063 ceph-mon[28006]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:37:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:37:58 smithi070 ceph-mon[32820]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:00.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:00 smithi114 ceph-mon[32988]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:00.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T18:38:00.830 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:00 smithi063 ceph-mon[28006]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T18:38:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:00 smithi070 ceph-mon[32820]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T18:38:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:01.795 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:01 smithi114 ceph-mon[32988]: Deploying daemon osd.5 on smithi114 2024-04-25T18:38:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:01 smithi063 ceph-mon[28006]: Deploying daemon osd.5 on smithi114 2024-04-25T18:38:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:01 smithi070 ceph-mon[32820]: Deploying daemon osd.5 on smithi114 2024-04-25T18:38:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:02 smithi063 ceph-mon[28006]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:02.952 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:02 smithi114 ceph-mon[32988]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:02 smithi070 ceph-mon[32820]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:04.828 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:04 smithi114 ceph-mon[32988]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:04 smithi063 ceph-mon[28006]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:04 smithi070 ceph-mon[32820]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:05.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:05.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:05.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:05 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:05.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:05 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:05.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:05 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:05.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:05 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:06.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:06.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:06.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:05 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:06.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:06 smithi114 ceph-mon[32988]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:06 smithi063 ceph-mon[28006]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:06.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:06 smithi070 ceph-mon[32820]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:08.573 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 5 on host 'smithi114' 2024-04-25T18:38:08.574 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.572+0000 7f845f7fe700 1 -- 172.21.15.114:0/1149357643 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8468062e80 con 0x7f84540a3cc0 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 msgr2=0x7f84540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 0x7f84540a6180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f846819a120 tx=0x7f84580096c0 comp rx=0 tx=0).stop 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 msgr2=0x7f8468199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f846400b530 tx=0x7f846400b840 comp rx=0 tx=0).stop 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 shutdown_connections 2024-04-25T18:38:08.576 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f84540a3cc0 0x7f84540a6180 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8468104640 0x7f84681a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8468102aa0 0x7f8468198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 --2- 172.21.15.114:0/1149357643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8468103ca0 0x7f8468199180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.575+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 >> 172.21.15.114:0/1149357643 conn(0x7f84680fe040 msgr2=0x7f8468107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.576+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 shutdown_connections 2024-04-25T18:38:08.577 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:38:08.576+0000 7f84706a3700 1 -- 172.21.15.114:0/1149357643 wait complete. 2024-04-25T18:38:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:08 smithi114 ceph-mon[32988]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:08 smithi063 ceph-mon[28006]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:08 smithi070 ceph-mon[32820]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.400 DEBUG:teuthology.orchestra.run.smithi114:osd.5> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T18:38:09.402 INFO:tasks.cephadm:Deploying osd.6 on smithi063 with /dev/vg_nvme/lv_2... 2024-04-25T18:38:09.403 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T18:38:09.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:09 smithi114 ceph-mon[32988]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:09 smithi114 ceph-mon[32988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.679 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:09 smithi063 ceph-mon[28006]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:09 smithi063 ceph-mon[28006]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.886 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:38:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:09 smithi070 ceph-mon[32820]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:09 smithi070 ceph-mon[32820]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T18:38:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:09.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T18:38:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T18:38:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:10 smithi114 ceph-mon[32988]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:10.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:10.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T18:38:10.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:10.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T18:38:10.911 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:10.912 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:10 smithi063 ceph-mon[28006]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T18:38:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: from='osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: osdmap e38: 6 total, 5 up, 6 in 2024-04-25T18:38:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:10 smithi070 ceph-mon[32820]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: from='osd.5 ' entity='osd.5' 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: osdmap e40: 6 total, 5 up, 6 in 2024-04-25T18:38:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:11.818 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T18:38:11.533+0000 7f0ea1826700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T18:38:11.818 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 18:38:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T18:38:11.679+0000 7f0e99611700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: from='osd.5 ' entity='osd.5' 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: osdmap e40: 6 total, 5 up, 6 in 2024-04-25T18:38:11.818 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:11.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:38:11.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: osdmap e39: 6 total, 5 up, 6 in 2024-04-25T18:38:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: from='osd.5 ' entity='osd.5' 2024-04-25T18:38:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: osdmap e40: 6 total, 5 up, 6 in 2024-04-25T18:38:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] boot 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T18:38:12.875 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] boot 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T18:38:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: osd.5 [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] boot 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: osdmap e41: 6 total, 6 up, 6 in 2024-04-25T18:38:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:38:13.206 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T18:38:13.207 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T18:38:13.207 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T18:38:13.207 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T18:38:13.207 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0356217 s, 294 MB/s 2024-04-25T18:38:13.207 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T18:38:14.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi114 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:13 smithi063 ceph-mon[28006]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T18:38:14.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi114 2024-04-25T18:38:14.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:13 smithi070 ceph-mon[32820]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T18:38:14.272 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi114 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:14.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:13 smithi114 ceph-mon[32988]: osdmap e42: 6 total, 6 up, 6 in 2024-04-25T18:38:14.717 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:38:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:14 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-25T18:38:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:14 smithi063 ceph-mon[28006]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:14 smithi063 ceph-mon[28006]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T18:38:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:14 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-25T18:38:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:14 smithi070 ceph-mon[32820]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:14 smithi070 ceph-mon[32820]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T18:38:15.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:14 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 8544M 2024-04-25T18:38:15.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:14 smithi114 ceph-mon[32988]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-04-25T18:38:15.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:14 smithi114 ceph-mon[32988]: osdmap e43: 6 total, 6 up, 6 in 2024-04-25T18:38:16.794 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:16 smithi063 ceph-mon[28006]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:16.794 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:16 smithi063 ceph-mon[28006]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T18:38:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:16 smithi114 ceph-mon[32988]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:16 smithi114 ceph-mon[32988]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T18:38:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:16 smithi070 ceph-mon[32820]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:16 smithi070 ceph-mon[32820]: osdmap e44: 6 total, 6 up, 6 in 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.301+0000 7f3409200700 1 -- 172.21.15.63:0/4042527518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3404108fd0 msgr2=0x7f34041093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.301+0000 7f3409200700 1 --2- 172.21.15.63:0/4042527518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3404108fd0 0x7f34041093f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f33f40099c0 tx=0x7f33f4009cd0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 -- 172.21.15.63:0/4042527518 shutdown_connections 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 --2- 172.21.15.63:0/4042527518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 0x7f3404111420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 --2- 172.21.15.63:0/4042527518 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3404108fd0 0x7f34041093f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 --2- 172.21.15.63:0/4042527518 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f340410a1d0 0x7f340410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 -- 172.21.15.63:0/4042527518 >> 172.21.15.63:0/4042527518 conn(0x7f3404077fa0 msgr2=0x7f340407a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:38:17.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 -- 172.21.15.63:0/4042527518 shutdown_connections 2024-04-25T18:38:17.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.302+0000 7f3409200700 1 -- 172.21.15.63:0/4042527518 wait complete. 2024-04-25T18:38:17.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 Processor -- start 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 -- start start 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 0x7f340419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 0x7f34041a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3404114aa0 con 0x7f340410a1d0 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3404114920 con 0x7f340410ab70 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.303+0000 7f3409200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3404114c20 con 0x7f3404108fd0 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:17.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41788/0 (socket says 172.21.15.63:41788) 2024-04-25T18:38:17.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 -- 172.21.15.63:0/3502530436 learned_addr learned my addr 172.21.15.63:0/3502530436 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:38:17.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3402a00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 0x7f340419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:17.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403e00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 0x7f34041a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 -- 172.21.15.63:0/3502530436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 msgr2=0x7f34041a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 0x7f34041a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 -- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 msgr2=0x7f340419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 0x7f340419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3403400700 1 -- 172.21.15.63:0/3502530436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34041a6da0 con 0x7f3404108fd0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.304+0000 7f3402a00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 0x7f340419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f3403400700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33f400b580 tx=0x7f33f4003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4009e60 con 0x7f3404108fd0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33f4005570 con 0x7f3404108fd0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34041a7030 con 0x7f3404108fd0 2024-04-25T18:38:17.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f400fe00 con 0x7f3404108fd0 2024-04-25T18:38:17.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.305+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34041a7480 con 0x7f3404108fd0 2024-04-25T18:38:17.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.306+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3404069730 con 0x7f3404108fd0 2024-04-25T18:38:17.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.308+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f33f400f4f0 con 0x7f3404108fd0 2024-04-25T18:38:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.310+0000 7f33fbe00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 0x7f33e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.310+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 4533+0+0 (secure 0 0 0) 0x7f33f40d0390 con 0x7f3404108fd0 2024-04-25T18:38:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.310+0000 7f3402a00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 0x7f33e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:17.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.311+0000 7f3402a00700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 0x7f33e80a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33f0011490 tx=0x7f33f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:38:17.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.314+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33f4013070 con 0x7f3404108fd0 2024-04-25T18:38:17.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:17.561+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3404062e80 con 0x7f33e80a3c40 2024-04-25T18:38:17.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:17 smithi114 ceph-mon[32988]: osdmap e45: 6 total, 6 up, 6 in 2024-04-25T18:38:17.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:17 smithi063 ceph-mon[28006]: osdmap e45: 6 total, 6 up, 6 in 2024-04-25T18:38:17.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:17 smithi070 ceph-mon[32820]: osdmap e45: 6 total, 6 up, 6 in 2024-04-25T18:38:18.782 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:18 smithi063 ceph-mon[28006]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:38:18.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:18 smithi063 ceph-mon[28006]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:18.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:18.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:18.783 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:18 smithi114 ceph-mon[32988]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:38:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:18 smithi114 ceph-mon[32988]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:18.855 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:18 smithi070 ceph-mon[32820]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T18:38:18.855 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:18 smithi070 ceph-mon[32820]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:18.855 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:18.855 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:18.855 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:20.693 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:20 smithi063 ceph-mon[28006]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T18:38:20.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:20 smithi114 ceph-mon[32988]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T18:38:20.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:20 smithi070 ceph-mon[32820]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-25T18:38:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:22 smithi063 ceph-mon[28006]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T18:38:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:22 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]: dispatch 2024-04-25T18:38:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:22 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]': finished 2024-04-25T18:38:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:22 smithi063 ceph-mon[28006]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T18:38:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:22 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:22 smithi114 ceph-mon[32988]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T18:38:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:22 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]: dispatch 2024-04-25T18:38:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:22 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]': finished 2024-04-25T18:38:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:22 smithi114 ceph-mon[32988]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T18:38:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:22 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:22 smithi070 ceph-mon[32820]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-25T18:38:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:22 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]: dispatch 2024-04-25T18:38:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:22 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4136244234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ca98c42c-8187-4097-a802-137756444990"}]': finished 2024-04-25T18:38:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:22 smithi070 ceph-mon[32820]: osdmap e46: 7 total, 6 up, 7 in 2024-04-25T18:38:22.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:22 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:23.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:23 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1423688016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:23.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:23 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1423688016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:23.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:23 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1423688016' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:24.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:24 smithi114 ceph-mon[32988]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:38:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:24 smithi063 ceph-mon[28006]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:38:24.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:24 smithi070 ceph-mon[32820]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:38:26.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:26 smithi114 ceph-mon[32988]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:38:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:26 smithi063 ceph-mon[28006]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:38:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:26 smithi070 ceph-mon[32820]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:38:28.853 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:28 smithi063 ceph-mon[28006]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:28 smithi070 ceph-mon[32820]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:29.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:28 smithi114 ceph-mon[32988]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:30.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:30 smithi063 ceph-mon[28006]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:30.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:30 smithi070 ceph-mon[32820]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:31.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:30 smithi114 ceph-mon[32988]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:32 smithi063 ceph-mon[28006]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T18:38:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:32 smithi070 ceph-mon[32820]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T18:38:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:32 smithi114 ceph-mon[32988]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T18:38:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:33.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:33 smithi063 ceph-mon[28006]: Deploying daemon osd.6 on smithi063 2024-04-25T18:38:33.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:33 smithi070 ceph-mon[32820]: Deploying daemon osd.6 on smithi063 2024-04-25T18:38:34.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:33 smithi114 ceph-mon[32988]: Deploying daemon osd.6 on smithi063 2024-04-25T18:38:34.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:34 smithi063 ceph-mon[28006]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:34.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:34 smithi070 ceph-mon[32820]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:35.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:34 smithi114 ceph-mon[32988]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:36 smithi063 ceph-mon[28006]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:36 smithi070 ceph-mon[32820]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:37.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:36 smithi114 ceph-mon[32988]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:38.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:38 smithi114 ceph-mon[32988]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:38 smithi063 ceph-mon[28006]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:38 smithi070 ceph-mon[32820]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:40.789 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:40 smithi063 ceph-mon[28006]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:40 smithi070 ceph-mon[32820]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:40 smithi114 ceph-mon[32988]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:41.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:41.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:41.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:42.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:42.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:42.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:42.765 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:42 smithi063 ceph-mon[28006]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:42 smithi070 ceph-mon[32820]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:43.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:42 smithi114 ceph-mon[32988]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:44 smithi063 ceph-mon[28006]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:44 smithi070 ceph-mon[32820]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:45.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:44 smithi114 ceph-mon[32988]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:45.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:45 smithi063 ceph-mon[28006]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T18:38:45.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:45 smithi070 ceph-mon[32820]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T18:38:46.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:45 smithi114 ceph-mon[32988]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T18:38:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T18:38:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T18:38:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:38:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:46.977 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:46.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.975+0000 7f33fbe00700 1 -- 172.21.15.63:0/3502530436 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3404062e80 con 0x7f33e80a3c40 2024-04-25T18:38:46.978 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 6 on host 'smithi063' 2024-04-25T18:38:46.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 msgr2=0x7f33e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:46.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 0x7f33e80a6100 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f33f0011490 tx=0x7f33f0009260 comp rx=0 tx=0).stop 2024-04-25T18:38:46.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 msgr2=0x7f340419f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f33f400b580 tx=0x7f33f4003af0 comp rx=0 tx=0).stop 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 shutdown_connections 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f33e80a3c40 0x7f33e80a6100 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3404108fd0 0x7f340419f140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f340410ab70 0x7f34041a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 --2- 172.21.15.63:0/3502530436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f340410a1d0 0x7f340419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 >> 172.21.15.63:0/3502530436 conn(0x7f3404077fa0 msgr2=0x7f340410d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 shutdown_connections 2024-04-25T18:38:46.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:38:46.979+0000 7f3409200700 1 -- 172.21.15.63:0/3502530436 wait complete. 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: osdmap e47: 7 total, 6 up, 7 in 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:47.869 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:47 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:47.927 DEBUG:teuthology.orchestra.run.smithi063:osd.6> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T18:38:47.931 INFO:tasks.cephadm:Deploying osd.7 on smithi070 with /dev/vg_nvme/lv_2... 2024-04-25T18:38:47.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:47 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:48.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: from='osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:38:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: osdmap e48: 7 total, 6 up, 7 in 2024-04-25T18:38:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:47 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:48.178 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:38:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:38:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:38:48.771 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] boot 2024-04-25T18:38:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T18:38:48.772 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] boot 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T18:38:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:49.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:38:49.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:38:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-04-25T18:38:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: osd.6 [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] boot 2024-04-25T18:38:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: osdmap e49: 7 total, 7 up, 7 in 2024-04-25T18:38:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:38:50.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:49 smithi063 ceph-mon[28006]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T18:38:50.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:49 smithi063 ceph-mon[28006]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00875781 s, 1.2 GB/s 2024-04-25T18:38:50.173 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-25T18:38:50.196 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:49 smithi070 ceph-mon[32820]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T18:38:50.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:49 smithi070 ceph-mon[32820]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:50.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:49 smithi114 ceph-mon[32988]: osdmap e50: 7 total, 7 up, 7 in 2024-04-25T18:38:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:49 smithi114 ceph-mon[32988]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:50.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-04-25T18:38:51.119 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:38:52.151 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:52 smithi070 ceph-mon[32820]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:52.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:52 smithi063 ceph-mon[28006]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:52 smithi114 ceph-mon[32988]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 -- 172.21.15.70:0/3259997689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 msgr2=0x7fc238101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 --2- 172.21.15.70:0/3259997689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238101dc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2340099c0 tx=0x7fc234009cd0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 -- 172.21.15.70:0/3259997689 shutdown_connections 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 --2- 172.21.15.70:0/3259997689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 0x7fc238109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 --2- 172.21.15.70:0/3259997689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238101dc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 --2- 172.21.15.70:0/3259997689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc238102ba0 0x7fc238103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.613+0000 7fc2401f9700 1 -- 172.21.15.70:0/3259997689 >> 172.21.15.70:0/3259997689 conn(0x7fc2380fcf40 msgr2=0x7fc2380ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- 172.21.15.70:0/3259997689 shutdown_connections 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- 172.21.15.70:0/3259997689 wait complete. 2024-04-25T18:38:52.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 Processor -- start 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- start start 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc238102ba0 0x7fc238199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 0x7fc2381a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d4b0 con 0x7fc238102ba0 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d330 con 0x7fc2381019a0 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc2401f9700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d630 con 0x7fc238103540 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.614+0000 7fc23df95700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47066/0 (socket says 172.21.15.70:47066) 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 -- 172.21.15.70:0/1756274185 learned_addr learned my addr 172.21.15.70:0/1756274185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23e796700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 0x7fc2381a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:52.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 -- 172.21.15.70:0/1756274185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 msgr2=0x7fc2381a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 0x7fc2381a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 -- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc238102ba0 msgr2=0x7fc238199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc238102ba0 0x7fc238199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 -- 172.21.15.70:0/1756274185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2381a08c0 con 0x7fc2381019a0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc23df95700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2340038c0 tx=0x7fc2340047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234027070 con 0x7fc2381019a0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2381a0b50 con 0x7fc2381019a0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc234004160 con 0x7fc2381019a0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234022e20 con 0x7fc2381019a0 2024-04-25T18:38:52.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.615+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2381a1080 con 0x7fc2381019a0 2024-04-25T18:38:52.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.616+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc238069730 con 0x7fc2381019a0 2024-04-25T18:38:52.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.617+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc23402a070 con 0x7fc2381019a0 2024-04-25T18:38:52.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.618+0000 7fc22effd700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 0x7fc2240a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:38:52.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.618+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 4965+0+0 (secure 0 0 0) 0x7fc2340d03a0 con 0x7fc2381019a0 2024-04-25T18:38:52.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.619+0000 7fc23d794700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 0x7fc2240a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:38:52.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.619+0000 7fc23d794700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 0x7fc2240a6180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2280097b0 tx=0x7fc228006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:38:52.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.622+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc234013070 con 0x7fc2381019a0 2024-04-25T18:38:52.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:38:52.829+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc2380ff190 con 0x7fc2240a3cc0 2024-04-25T18:38:53.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:53 smithi063 ceph-mon[28006]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:53.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:53.300 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:53.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:53.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:53 smithi070 ceph-mon[32820]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:53.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:53.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:53.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:53.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:53 smithi114 ceph-mon[32988]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:38:53.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:38:53.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:38:53.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi063 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi063 2024-04-25T18:38:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi063 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:38:54.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/2273225755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]': finished 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:38:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:56 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/926953251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:56.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/2273225755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]': finished 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:38:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:56 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/926953251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/2273225755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]: dispatch 2024-04-25T18:38:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "164885fd-3ba5-4de5-9224-0de701470a27"}]': finished 2024-04-25T18:38:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: osdmap e51: 8 total, 7 up, 8 in 2024-04-25T18:38:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:38:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:56 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/926953251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:38:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:38:58 smithi070 ceph-mon[32820]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:58.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:38:58 smithi114 ceph-mon[32988]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:38:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:38:58 smithi063 ceph-mon[28006]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:00.691 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:00 smithi070 ceph-mon[32820]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:00.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:00 smithi114 ceph-mon[32988]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:00 smithi063 ceph-mon[28006]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:02 smithi070 ceph-mon[32820]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:02 smithi114 ceph-mon[32988]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:02 smithi063 ceph-mon[28006]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:03.622 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:03 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T18:39:03.623 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:03 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.623 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:03 smithi070 ceph-mon[32820]: Deploying daemon osd.7 on smithi070 2024-04-25T18:39:03.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:03 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T18:39:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:03 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:03 smithi114 ceph-mon[32988]: Deploying daemon osd.7 on smithi070 2024-04-25T18:39:03.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:03 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T18:39:03.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:03 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:03.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:03 smithi063 ceph-mon[28006]: Deploying daemon osd.7 on smithi070 2024-04-25T18:39:04.571 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:04 smithi070 ceph-mon[32820]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:04.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:04 smithi114 ceph-mon[32988]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:04 smithi063 ceph-mon[28006]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:06.737 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:06 smithi070 ceph-mon[32820]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:06 smithi114 ceph-mon[32988]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:06 smithi063 ceph-mon[28006]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:08 smithi114 ceph-mon[32988]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:08 smithi063 ceph-mon[28006]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:08.942 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:08 smithi070 ceph-mon[32820]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:08.942 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.942 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:08.942 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:10.668 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:10 smithi070 ceph-mon[32820]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:10.802 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:10 smithi114 ceph-mon[32988]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:10 smithi063 ceph-mon[28006]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:11.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.865+0000 7fc22effd700 1 -- 172.21.15.70:0/1756274185 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc2380ff190 con 0x7fc2240a3cc0 2024-04-25T18:39:11.867 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 7 on host 'smithi070' 2024-04-25T18:39:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 msgr2=0x7fc2240a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 0x7fc2240a6180 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2280097b0 tx=0x7fc228006d20 comp rx=0 tx=0).stop 2024-04-25T18:39:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 msgr2=0x7fc238198d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:11.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc2340038c0 tx=0x7fc2340047b0 comp rx=0 tx=0).stop 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 shutdown_connections 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc2240a3cc0 0x7fc2240a6180 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc238103540 0x7fc2381a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2381019a0 0x7fc238198d10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 --2- 172.21.15.70:0/1756274185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc238102ba0 0x7fc238199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 >> 172.21.15.70:0/1756274185 conn(0x7fc2380fcf40 msgr2=0x7fc2380fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 shutdown_connections 2024-04-25T18:39:11.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:39:11.867+0000 7fc2401f9700 1 -- 172.21.15.70:0/1756274185 wait complete. 2024-04-25T18:39:12.133 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.133 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.133 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:12.133 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:12.133 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.134 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.134 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:12.134 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:12.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:12.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:12.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:12.506 DEBUG:teuthology.orchestra.run.smithi070:osd.7> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T18:39:12.509 INFO:tasks.cephadm:Deploying osd.8 on smithi114 with /dev/vg_nvme/lv_2... 2024-04-25T18:39:12.509 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T18:39:12.750 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:39:13.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:12 smithi114 ceph-mon[32988]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:13.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:12 smithi114 ceph-mon[32988]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:13.134 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:12 smithi114 ceph-mon[32988]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:13.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:12 smithi070 ceph-mon[32820]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:13.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:12 smithi070 ceph-mon[32820]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:13.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:12 smithi070 ceph-mon[32820]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:13.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:12 smithi063 ceph-mon[28006]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:13.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:12 smithi063 ceph-mon[28006]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:13.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:12 smithi063 ceph-mon[28006]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T18:39:14.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T18:39:14.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T18:39:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:13 smithi070 ceph-mon[32820]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:14.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T18:39:14.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T18:39:14.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:14.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:13 smithi114 ceph-mon[32988]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T18:39:14.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: from='osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: osdmap e52: 8 total, 7 up, 8 in 2024-04-25T18:39:14.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:14.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:39:14.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:13 smithi063 ceph-mon[28006]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150623 s, 696 MB/s 2024-04-25T18:39:14.679 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-25T18:39:14.953 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T18:39:14.627+0000 7f55f9d64700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T18:39:14.954 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T18:39:14.637+0000 7f55f134e700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:39:15.286 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: osdmap e53: 8 total, 7 up, 8 in 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: from='osd.7 ' entity='osd.7' 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] boot 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T18:39:15.287 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.313 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2024-04-25T18:39:15.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: osdmap e53: 8 total, 7 up, 8 in 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: from='osd.7 ' entity='osd.7' 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] boot 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T18:39:15.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: osdmap e53: 8 total, 7 up, 8 in 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: from='osd.7 ' entity='osd.7' 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: osd.7 [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] boot 2024-04-25T18:39:15.448 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: osdmap e54: 8 total, 8 up, 8 in 2024-04-25T18:39:15.449 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:39:15.547 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi070 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:16 smithi114 ceph-mon[32988]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T18:39:16.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi070 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:16 smithi063 ceph-mon[28006]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T18:39:16.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:39:16.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi070 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 5696M 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:16 smithi070 ceph-mon[32820]: osdmap e55: 8 total, 8 up, 8 in 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 -- 172.21.15.114:0/2790764778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30102b90 msgr2=0x7f8d30102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 --2- 172.21.15.114:0/2790764778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30102b90 0x7f8d30102ff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d20009990 tx=0x7f8d20009ca0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 -- 172.21.15.114:0/2790764778 shutdown_connections 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 --2- 172.21.15.114:0/2790764778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30103530 0x7f8d30109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 --2- 172.21.15.114:0/2790764778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30101990 0x7f8d30101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.242 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 --2- 172.21.15.114:0/2790764778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30102b90 0x7f8d30102ff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 -- 172.21.15.114:0/2790764778 >> 172.21.15.114:0/2790764778 conn(0x7f8d300fcf30 msgr2=0x7f8d300ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:39:17.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 -- 172.21.15.114:0/2790764778 shutdown_connections 2024-04-25T18:39:17.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.240+0000 7f8d353d6700 1 -- 172.21.15.114:0/2790764778 wait complete. 2024-04-25T18:39:17.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 Processor -- start 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 -- start start 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30101990 0x7f8d30105920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102b90 0x7f8d30103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d520 con 0x7f8d30102b90 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d3a0 con 0x7f8d30101990 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d353d6700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3010d6a0 con 0x7f8d30103530 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:38200/0 (socket says 172.21.15.114:38200) 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 -- 172.21.15.114:0/3024569230 learned_addr learned my addr 172.21.15.114:0/3024569230 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2effd700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30101990 0x7f8d30105920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:17.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 -- 172.21.15.114:0/3024569230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30101990 msgr2=0x7f8d30105920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30101990 0x7f8d30105920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 -- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102b90 msgr2=0x7f8d30103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102b90 0x7f8d30103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.241+0000 7f8d2f7fe700 1 -- 172.21.15.114:0/3024569230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d30104d60 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d2f7fe700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800b970 tx=0x7f8d2800bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28019070 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d2800cde0 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d301a2e60 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28007370 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.242+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d301a3340 con 0x7f8d30103530 2024-04-25T18:39:17.246 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.243+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d30069730 con 0x7f8d30103530 2024-04-25T18:39:17.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.244+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d2801c070 con 0x7f8d30103530 2024-04-25T18:39:17.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.244+0000 7f8d17fff700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:17.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.245+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5426+0+0 (secure 0 0 0) 0x7f8d280cf480 con 0x7f8d30103530 2024-04-25T18:39:17.247 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.245+0000 7f8d2effd700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:17.248 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.246+0000 7f8d2effd700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 0x7f8d180a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d301029f0 tx=0x7f8d24009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:39:17.249 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.248+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d28023070 con 0x7f8d30103530 2024-04-25T18:39:17.457 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:17.455+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8d3006b6a0 con 0x7f8d180a3c40 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:18.257 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:18.269 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T18:39:18.269 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:18.269 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:18.270 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:18.270 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:18.270 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:17 smithi063 ceph-mon[28006]: osdmap e56: 8 total, 8 up, 8 in 2024-04-25T18:39:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:17 smithi063 ceph-mon[28006]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:18 smithi063 ceph-mon[28006]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:18.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:19.170 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:19 smithi114 ceph-mon[32988]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T18:39:19.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:19 smithi070 ceph-mon[32820]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T18:39:19.279 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:19 smithi063 ceph-mon[28006]: osdmap e57: 8 total, 8 up, 8 in 2024-04-25T18:39:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:20 smithi114 ceph-mon[32988]: osdmap e58: 8 total, 8 up, 8 in 2024-04-25T18:39:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:20 smithi114 ceph-mon[32988]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:20 smithi063 ceph-mon[28006]: osdmap e58: 8 total, 8 up, 8 in 2024-04-25T18:39:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:20 smithi063 ceph-mon[28006]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:20.450 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:20 smithi070 ceph-mon[32820]: osdmap e58: 8 total, 8 up, 8 in 2024-04-25T18:39:20.451 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:20 smithi070 ceph-mon[32820]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/1267086546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]': finished 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:21 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/4159308256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:21.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/1267086546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]': finished 2024-04-25T18:39:21.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T18:39:21.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:21.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:21 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/4159308256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:21.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/1267086546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]: dispatch 2024-04-25T18:39:21.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "820c8864-3156-4c9f-b583-0360fa2455eb"}]': finished 2024-04-25T18:39:21.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: osdmap e59: 9 total, 8 up, 9 in 2024-04-25T18:39:21.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:21.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:21 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/4159308256' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:22.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:22 smithi114 ceph-mon[32988]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:22 smithi063 ceph-mon[28006]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:22 smithi070 ceph-mon[32820]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:24 smithi063 ceph-mon[28006]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:24 smithi070 ceph-mon[32820]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:24.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:24 smithi114 ceph-mon[32988]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:26.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:26 smithi063 ceph-mon[28006]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:26.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:26 smithi070 ceph-mon[32820]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:26.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:26 smithi114 ceph-mon[32988]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:28.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:28 smithi063 ceph-mon[28006]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:28.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:28 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T18:39:28.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:28 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:28.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:28 smithi063 ceph-mon[28006]: Deploying daemon osd.8 on smithi114 2024-04-25T18:39:28.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:28 smithi070 ceph-mon[32820]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:28.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:28 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T18:39:28.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:28 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:28.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:28 smithi070 ceph-mon[32820]: Deploying daemon osd.8 on smithi114 2024-04-25T18:39:28.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:28 smithi114 ceph-mon[32988]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:28.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:28 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T18:39:28.517 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:28 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:28.518 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:28 smithi114 ceph-mon[32988]: Deploying daemon osd.8 on smithi114 2024-04-25T18:39:30.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:30 smithi070 ceph-mon[32820]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:30.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:30 smithi114 ceph-mon[32988]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:30 smithi063 ceph-mon[28006]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:32.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:32 smithi070 ceph-mon[32820]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:32.471 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:32 smithi114 ceph-mon[32988]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:32 smithi063 ceph-mon[28006]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:34 smithi114 ceph-mon[32988]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:34 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:34 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:34 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:34 smithi070 ceph-mon[32820]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:34 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:34 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:34 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:34 smithi063 ceph-mon[28006]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:34 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:34 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:34 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:36.427 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:36 smithi114 ceph-mon[32988]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:36.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:36 smithi070 ceph-mon[32820]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:36 smithi063 ceph-mon[28006]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:37.149 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 8 on host 'smithi114' 2024-04-25T18:39:37.149 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.148+0000 7f8d17fff700 1 -- 172.21.15.114:0/3024569230 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8d3006b6a0 con 0x7f8d180a3c40 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 msgr2=0x7f8d180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 0x7f8d180a6100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d301029f0 tx=0x7f8d24009190 comp rx=0 tx=0).stop 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 msgr2=0x7f8d301044b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800b970 tx=0x7f8d2800bc80 comp rx=0 tx=0).stop 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 shutdown_connections 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f8d180a3c40 0x7f8d180a6100 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8d30103530 0x7f8d301044b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:37.151 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d30101990 0x7f8d30105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 --2- 172.21.15.114:0/3024569230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d30102b90 0x7f8d30103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.150+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 >> 172.21.15.114:0/3024569230 conn(0x7f8d300fcf30 msgr2=0x7f8d3006af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:39:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.151+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 shutdown_connections 2024-04-25T18:39:37.152 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:39:37.151+0000 7f8d353d6700 1 -- 172.21.15.114:0/3024569230 wait complete. 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:37 smithi114 ceph-mon[32988]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:37 smithi063 ceph-mon[28006]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:37 smithi070 ceph-mon[32820]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T18:39:37.820 DEBUG:teuthology.orchestra.run.smithi114:osd.8> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T18:39:37.822 INFO:tasks.cephadm:Deploying osd.9 on smithi063 with /dev/vg_nvme/lv_1... 2024-04-25T18:39:37.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T18:39:38.274 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:39:38.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T18:39:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T18:39:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:38 smithi114 ceph-mon[32988]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:38 smithi063 ceph-mon[28006]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T18:39:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: from='osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: osdmap e60: 9 total, 8 up, 9 in 2024-04-25T18:39:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:38 smithi070 ceph-mon[32820]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:39:40.259 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:40 smithi063 ceph-mon[28006]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:40.259 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:40 smithi063 ceph-mon[28006]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:39:40.259 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:40 smithi063 ceph-mon[28006]: osdmap e61: 9 total, 8 up, 9 in 2024-04-25T18:39:40.259 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:40.259 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:40 smithi063 ceph-mon[28006]: from='osd.8 ' entity='osd.8' 2024-04-25T18:39:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-mon[32988]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-mon[32988]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:39:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-mon[32988]: osdmap e61: 9 total, 8 up, 9 in 2024-04-25T18:39:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-mon[32988]: from='osd.8 ' entity='osd.8' 2024-04-25T18:39:40.318 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T18:39:40.044+0000 7f17fac0d700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T18:39:40.318 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 18:39:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T18:39:40.173+0000 7f17f21f7700 -1 osd.8 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:39:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:40 smithi070 ceph-mon[32820]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:40 smithi070 ceph-mon[32820]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:39:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:40 smithi070 ceph-mon[32820]: osdmap e61: 9 total, 8 up, 9 in 2024-04-25T18:39:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:40.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:40 smithi070 ceph-mon[32820]: from='osd.8 ' entity='osd.8' 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: osdmap e62: 9 total, 8 up, 9 in 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.207 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] boot 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T18:39:41.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: osdmap e62: 9 total, 8 up, 9 in 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] boot 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T18:39:41.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:39:41.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:39:41.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: osdmap e62: 9 total, 8 up, 9 in 2024-04-25T18:39:41.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: osd.8 [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] boot 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: osdmap e63: 9 total, 9 up, 9 in 2024-04-25T18:39:41.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0322614 s, 325 MB/s 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 2024-04-25T18:39:41.586 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T18:39:42.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:42 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi114 2024-04-25T18:39:42.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:42 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-25T18:39:42.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:42 smithi063 ceph-mon[28006]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:42 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi114 2024-04-25T18:39:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:42 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-25T18:39:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:42 smithi070 ceph-mon[32820]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:42 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi114 2024-04-25T18:39:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:42 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 5696M 2024-04-25T18:39:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:42 smithi114 ceph-mon[32988]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-25T18:39:42.603 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_1 2024-04-25T18:39:43.058 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:39:43.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:43 smithi063 ceph-mon[28006]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T18:39:43.399 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:43 smithi063 ceph-mon[28006]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T18:39:43.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:43 smithi070 ceph-mon[32820]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T18:39:43.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:43 smithi070 ceph-mon[32820]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T18:39:43.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:43 smithi114 ceph-mon[32988]: osdmap e64: 9 total, 9 up, 9 in 2024-04-25T18:39:43.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:43 smithi114 ceph-mon[32988]: osdmap e65: 9 total, 9 up, 9 in 2024-04-25T18:39:44.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:44 smithi114 ceph-mon[32988]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:44 smithi114 ceph-mon[32988]: osdmap e66: 9 total, 9 up, 9 in 2024-04-25T18:39:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:44 smithi063 ceph-mon[28006]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:44 smithi063 ceph-mon[28006]: osdmap e66: 9 total, 9 up, 9 in 2024-04-25T18:39:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:44 smithi070 ceph-mon[32820]: pgmap v229: 1 pgs: 1 peering; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:44.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:44 smithi070 ceph-mon[32820]: osdmap e66: 9 total, 9 up, 9 in 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 -- 172.21.15.63:0/2239503541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 msgr2=0x7f0e3c1082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2239503541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c1082b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009990 tx=0x7f0e28009ca0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 -- 172.21.15.63:0/2239503541 shutdown_connections 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2239503541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c103590 0x7f0e3c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2239503541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c1087f0 0x7f0e3c10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2239503541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c1082b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.540+0000 7f0e42a00700 1 -- 172.21.15.63:0/2239503541 >> 172.21.15.63:0/2239503541 conn(0x7f0e3c0fcf40 msgr2=0x7f0e3c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:39:45.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.541+0000 7f0e42a00700 1 -- 172.21.15.63:0/2239503541 shutdown_connections 2024-04-25T18:39:45.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.541+0000 7f0e42a00700 1 -- 172.21.15.63:0/2239503541 wait complete. 2024-04-25T18:39:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 Processor -- start 2024-04-25T18:39:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 -- start start 2024-04-25T18:39:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 0x7f0e3c19aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:45.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1087f0 0x7f0e3c1a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e550 con 0x7f0e3c105ec0 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e3d0 con 0x7f0e3c103590 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.542+0000 7f0e42a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c10e6d0 con 0x7f0e3c1087f0 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35504/0 (socket says 172.21.15.63:35504) 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 -- 172.21.15.63:0/2406450587 learned_addr learned my addr 172.21.15.63:0/2406450587 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3be00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 0x7f0e3c19aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e40a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1087f0 0x7f0e3c1a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 -- 172.21.15.63:0/2406450587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1087f0 msgr2=0x7f0e3c1a2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1087f0 0x7f0e3c1a2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 -- 172.21.15.63:0/2406450587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 msgr2=0x7f0e3c19aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:39:45.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 0x7f0e3c19aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:39:45.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3b400700 1 -- 172.21.15.63:0/2406450587 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c1a2b30 con 0x7f0e3c105ec0 2024-04-25T18:39:45.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.543+0000 7f0e3be00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 0x7f0e3c19aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:39:45.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e3b400700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000c00 tx=0x7f0e28009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:39:45.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28026070 con 0x7f0e3c105ec0 2024-04-25T18:39:45.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e28021410 con 0x7f0e3c105ec0 2024-04-25T18:39:45.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28017650 con 0x7f0e3c105ec0 2024-04-25T18:39:45.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c1a2dc0 con 0x7f0e3c105ec0 2024-04-25T18:39:45.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.544+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c1a31c0 con 0x7f0e3c105ec0 2024-04-25T18:39:45.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.545+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3c069730 con 0x7f0e3c105ec0 2024-04-25T18:39:45.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.547+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0e28029070 con 0x7f0e3c105ec0 2024-04-25T18:39:45.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.549+0000 7f0e38c00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 0x7f0e200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:39:45.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.549+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5829+0+0 (secure 0 0 0) 0x7f0e280d0ea0 con 0x7f0e3c105ec0 2024-04-25T18:39:45.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.549+0000 7f0e3be00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 0x7f0e200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:39:45.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.550+0000 7f0e3be00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 0x7f0e200a6180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c011490 tx=0x7f0e2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:39:45.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.552+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e28030070 con 0x7f0e3c105ec0 2024-04-25T18:39:45.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:39:45.798+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0e3c062e80 con 0x7f0e200a3cc0 2024-04-25T18:39:45.800 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:45 smithi063 ceph-mon[28006]: osdmap e67: 9 total, 9 up, 9 in 2024-04-25T18:39:45.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:45 smithi114 ceph-mon[32988]: osdmap e67: 9 total, 9 up, 9 in 2024-04-25T18:39:45.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:45 smithi070 ceph-mon[32820]: osdmap e67: 9 total, 9 up, 9 in 2024-04-25T18:39:46.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:46 smithi114 ceph-mon[32988]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-04-25T18:39:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:46 smithi114 ceph-mon[32988]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:46.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:46 smithi063 ceph-mon[28006]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-04-25T18:39:46.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:46 smithi063 ceph-mon[28006]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:46.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:46.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:46.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:46 smithi070 ceph-mon[32820]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-04-25T18:39:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:46 smithi070 ceph-mon[32820]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:39:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:39:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:39:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:39:48.764 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:48 smithi063 ceph-mon[28006]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T18:39:48.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:48 smithi114 ceph-mon[32988]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T18:39:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:48 smithi070 ceph-mon[32820]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T18:39:50.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:50 smithi063 ceph-mon[28006]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:39:50.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:50 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]: dispatch 2024-04-25T18:39:50.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:50 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]': finished 2024-04-25T18:39:50.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:50 smithi063 ceph-mon[28006]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T18:39:50.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:50 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:39:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:50 smithi070 ceph-mon[32820]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:39:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:50 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]: dispatch 2024-04-25T18:39:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:50 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]': finished 2024-04-25T18:39:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:50 smithi070 ceph-mon[32820]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T18:39:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:50 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:39:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:50 smithi114 ceph-mon[32988]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:39:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:50 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]: dispatch 2024-04-25T18:39:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:50 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1754318962' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5f01be5b-f17a-4fc6-a8cb-10630b362163"}]': finished 2024-04-25T18:39:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:50 smithi114 ceph-mon[32988]: osdmap e68: 10 total, 9 up, 10 in 2024-04-25T18:39:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:50 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:39:51.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:51 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1421154140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:51.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:51 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1421154140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:52.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:51 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1421154140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:39:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:52 smithi063 ceph-mon[28006]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:39:52.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:52 smithi070 ceph-mon[32820]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:39:53.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:52 smithi114 ceph-mon[32988]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-25T18:39:54.875 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:54 smithi063 ceph-mon[28006]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:39:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:54 smithi070 ceph-mon[32820]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:39:55.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:54 smithi114 ceph-mon[32988]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-25T18:39:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:56 smithi063 ceph-mon[28006]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:56.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:56 smithi070 ceph-mon[32820]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:57.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:56 smithi114 ceph-mon[32988]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:39:58 smithi063 ceph-mon[28006]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:39:58 smithi070 ceph-mon[32820]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:39:59.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:39:58 smithi114 ceph-mon[32988]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:00 smithi063 ceph-mon[28006]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:00 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T18:40:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T18:40:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:00 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:00 smithi070 ceph-mon[32820]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:00 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T18:40:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T18:40:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:00 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:01.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:00 smithi114 ceph-mon[32988]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:00 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T18:40:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T18:40:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:00 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:01 smithi063 ceph-mon[28006]: Deploying daemon osd.9 on smithi063 2024-04-25T18:40:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:01 smithi070 ceph-mon[32820]: Deploying daemon osd.9 on smithi063 2024-04-25T18:40:02.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:01 smithi114 ceph-mon[32988]: Deploying daemon osd.9 on smithi063 2024-04-25T18:40:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:02 smithi063 ceph-mon[28006]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:02 smithi070 ceph-mon[32820]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:03.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:02 smithi114 ceph-mon[32988]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:04 smithi063 ceph-mon[28006]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:04 smithi070 ceph-mon[32820]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:05.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:04 smithi114 ceph-mon[32988]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:06.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:06 smithi070 ceph-mon[32820]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:07.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:06 smithi114 ceph-mon[32988]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:07.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:06 smithi063 ceph-mon[28006]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:08 smithi070 ceph-mon[32820]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:09.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:08 smithi114 ceph-mon[32988]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:09.115 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:08 smithi063 ceph-mon[28006]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:10.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:10 smithi063 ceph-mon[28006]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:10.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:10.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:10.787 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:10 smithi114 ceph-mon[32988]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:10 smithi070 ceph-mon[32820]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:12.785 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:12 smithi063 ceph-mon[28006]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:12.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:12 smithi114 ceph-mon[32988]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:12 smithi070 ceph-mon[32820]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:13.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:13 smithi114 ceph-mon[32988]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T18:40:13.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:13 smithi063 ceph-mon[28006]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T18:40:13.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:13 smithi070 ceph-mon[32820]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: osdmap e69: 10 total, 9 up, 10 in 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:14 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T18:40:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: osdmap e69: 10 total, 9 up, 10 in 2024-04-25T18:40:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:40:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:14 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T18:40:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: osdmap e69: 10 total, 9 up, 10 in 2024-04-25T18:40:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T18:40:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:14 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.009 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 9 on host 'smithi063' 2024-04-25T18:40:15.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.007+0000 7f0e38c00700 1 -- 172.21.15.63:0/2406450587 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0e3c062e80 con 0x7f0e200a3cc0 2024-04-25T18:40:15.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.012+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 msgr2=0x7f0e200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.012+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 0x7f0e200a6180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c011490 tx=0x7f0e2c009260 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.012+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 msgr2=0x7f0e3c19b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.012+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0e28000c00 tx=0x7f0e28009580 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.012+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 shutdown_connections 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0e200a3cc0 0x7f0e200a6180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0e3c1087f0 0x7f0e3c1a2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e3c103590 0x7f0e3c19aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 --2- 172.21.15.63:0/2406450587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e3c105ec0 0x7f0e3c19b410 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:15.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 >> 172.21.15.63:0/2406450587 conn(0x7f0e3c0fcf40 msgr2=0x7f0e3c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:40:15.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 shutdown_connections 2024-04-25T18:40:15.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:40:15.013+0000 7f0e42a00700 1 -- 172.21.15.63:0/2406450587 wait complete. 2024-04-25T18:40:15.694 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:40:15.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: osdmap e70: 10 total, 9 up, 10 in 2024-04-25T18:40:15.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.695 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:40:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: osdmap e70: 10 total, 9 up, 10 in 2024-04-25T18:40:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.869 DEBUG:teuthology.orchestra.run.smithi063:osd.9> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T18:40:15.876 INFO:tasks.cephadm:Deploying osd.10 on smithi070 with /dev/vg_nvme/lv_1... 2024-04-25T18:40:15.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T18:40:15.902 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T18:40:15.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: osdmap e70: 10 total, 9 up, 10 in 2024-04-25T18:40:15.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:15.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:15.903 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:16.122 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:40:16.738 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:16 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:40:16.738 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:16 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:40:16.738 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:16 smithi063 ceph-mon[28006]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:16.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:16 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:16.739 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:16 smithi063 ceph-mon[28006]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' 2024-04-25T18:40:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:16 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:40:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:16 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:40:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:16 smithi114 ceph-mon[32988]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:16 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:16 smithi114 ceph-mon[32988]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' 2024-04-25T18:40:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:16 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:40:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:16 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:40:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:16 smithi070 ceph-mon[32820]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:16 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:16 smithi070 ceph-mon[32820]: from='osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816]' entity='osd.9' 2024-04-25T18:40:17.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:17.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:17 smithi114 ceph-mon[32988]: osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] boot 2024-04-25T18:40:17.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:17 smithi114 ceph-mon[32988]: osdmap e71: 10 total, 10 up, 10 in 2024-04-25T18:40:17.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:17 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:17.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:17.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:17 smithi063 ceph-mon[28006]: osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] boot 2024-04-25T18:40:17.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:17 smithi063 ceph-mon[28006]: osdmap e71: 10 total, 10 up, 10 in 2024-04-25T18:40:17.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:17 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:17.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:17.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:17 smithi070 ceph-mon[32820]: osd.9 [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] boot 2024-04-25T18:40:17.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:17 smithi070 ceph-mon[32820]: osdmap e71: 10 total, 10 up, 10 in 2024-04-25T18:40:17.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:17 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:40:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T18:40:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T18:40:18.110 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-04-25T18:40:18.110 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-04-25T18:40:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00856003 s, 1.2 GB/s 2024-04-25T18:40:18.111 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-04-25T18:40:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:18 smithi070 ceph-mon[32820]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:18 smithi070 ceph-mon[32820]: osdmap e72: 10 total, 10 up, 10 in 2024-04-25T18:40:18.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-04-25T18:40:19.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:18 smithi114 ceph-mon[32988]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:18 smithi114 ceph-mon[32988]: osdmap e72: 10 total, 10 up, 10 in 2024-04-25T18:40:19.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:18 smithi063 ceph-mon[28006]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail 2024-04-25T18:40:19.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:18 smithi063 ceph-mon[28006]: osdmap e72: 10 total, 10 up, 10 in 2024-04-25T18:40:19.145 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 -- 172.21.15.70:0/2307984153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 msgr2=0x7f9fd0103000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2307984153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0103000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009990 tx=0x7f9fc4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 -- 172.21.15.70:0/2307984153 shutdown_connections 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2307984153 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 0x7f9fd0101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2307984153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fd0103540 0x7f9fd0109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2307984153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0103000 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 -- 172.21.15.70:0/2307984153 >> 172.21.15.70:0/2307984153 conn(0x7f9fd00fcf40 msgr2=0x7f9fd00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:40:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 -- 172.21.15.70:0/2307984153 shutdown_connections 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 -- 172.21.15.70:0/2307984153 wait complete. 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.892+0000 7f9fd7834700 1 Processor -- start 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 -- start start 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 0x7f9fd0198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fd0103540 0x7f9fd01a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010e5b0 con 0x7f9fd0103540 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010e430 con 0x7f9fd0102ba0 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010e730 con 0x7f9fd01019a0 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33086/0 (socket says 172.21.15.70:33086) 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 -- 172.21.15.70:0/2179618454 learned_addr learned my addr 172.21.15.70:0/2179618454 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd55d0700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 0x7f9fd0198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 -- 172.21.15.70:0/2179618454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 msgr2=0x7f9fd0198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 0x7f9fd0198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 -- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fd0103540 msgr2=0x7f9fd01a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fd0103540 0x7f9fd01a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 -- 172.21.15.70:0/2179618454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd01a0890 con 0x7f9fd0102ba0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd4dcf700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006010 tx=0x7f9fc40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4027070 con 0x7f9fd0102ba0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fc4004080 con 0x7f9fd0102ba0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd01a0b20 con 0x7f9fd0102ba0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.893+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4022e00 con 0x7f9fd0102ba0 2024-04-25T18:40:20.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.894+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a0ff0 con 0x7f9fd0102ba0 2024-04-25T18:40:20.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.894+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd002df80 con 0x7f9fd0102ba0 2024-04-25T18:40:20.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.895+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9fc402a070 con 0x7f9fd0102ba0 2024-04-25T18:40:20.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.896+0000 7f9fce7fc700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 0x7f9fb80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:20.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.896+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 6261+0+0 (secure 0 0 0) 0x7f9fc40d0a50 con 0x7f9fd0102ba0 2024-04-25T18:40:20.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.897+0000 7f9fd55d0700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 0x7f9fb80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:20.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.898+0000 7f9fd55d0700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 0x7f9fb80a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009550 tx=0x7f9fc80091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:40:20.899 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:20 smithi070 ceph-mon[32820]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:20.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:20.900+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9fc4013070 con 0x7f9fd0102ba0 2024-04-25T18:40:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:20 smithi114 ceph-mon[32988]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:21.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:21.104+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9fd0062e80 con 0x7f9fb80a3c40 2024-04-25T18:40:21.121 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:20 smithi063 ceph-mon[28006]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:21.947 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:21.947 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:21.948 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:21.949 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:21 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:22.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:21 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:22.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:22.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:21 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:22.926 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:22 smithi070 ceph-mon[32820]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:22.926 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:22 smithi070 ceph-mon[32820]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:22.926 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:22 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi063 2024-04-25T18:40:22.926 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:22 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T18:40:23.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:22 smithi114 ceph-mon[32988]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:23.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:22 smithi114 ceph-mon[32988]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:23.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:22 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi063 2024-04-25T18:40:23.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:22 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T18:40:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:22 smithi063 ceph-mon[28006]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:22 smithi063 ceph-mon[28006]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:22 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi063 2024-04-25T18:40:23.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:22 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/3331469541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]': finished 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:24 smithi070 ceph-mon[32820]: from='client.? 172.21.15.70:0/1012924236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:25.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/3331469541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]': finished 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:24 smithi114 ceph-mon[32988]: from='client.? 172.21.15.70:0/1012924236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/3331469541' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:25.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]: dispatch 2024-04-25T18:40:25.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "067a5560-1dc0-45a8-8a1e-82061039239e"}]': finished 2024-04-25T18:40:25.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: osdmap e73: 11 total, 10 up, 11 in 2024-04-25T18:40:25.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:25.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:24 smithi063 ceph-mon[28006]: from='client.? 172.21.15.70:0/1012924236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:26 smithi070 ceph-mon[32820]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:27.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:26 smithi114 ceph-mon[32988]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:27.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:26 smithi063 ceph-mon[28006]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:28 smithi070 ceph-mon[32820]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:29.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:28 smithi114 ceph-mon[32988]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:28 smithi063 ceph-mon[28006]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:30.785 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:30 smithi070 ceph-mon[32820]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:31.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:30 smithi114 ceph-mon[32988]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:30 smithi063 ceph-mon[28006]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:31.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:31 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T18:40:31.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:31 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:32.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:31 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T18:40:32.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:31 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:32.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:31 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T18:40:32.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:31 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:32 smithi070 ceph-mon[32820]: Deploying daemon osd.10 on smithi070 2024-04-25T18:40:32.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:32 smithi070 ceph-mon[32820]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:32 smithi114 ceph-mon[32988]: Deploying daemon osd.10 on smithi070 2024-04-25T18:40:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:32 smithi114 ceph-mon[32988]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:33.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:32 smithi063 ceph-mon[28006]: Deploying daemon osd.10 on smithi070 2024-04-25T18:40:33.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:32 smithi063 ceph-mon[28006]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:34.818 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:34 smithi070 ceph-mon[32820]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:35.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:34 smithi114 ceph-mon[32988]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:34 smithi063 ceph-mon[28006]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:36 smithi070 ceph-mon[32820]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:36 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:36.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:36 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:36.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:36 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:37.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:36 smithi114 ceph-mon[32988]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:36 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:36 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:36 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:36 smithi063 ceph-mon[28006]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:36 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:38.974 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:38 smithi070 ceph-mon[32820]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:39.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:38 smithi114 ceph-mon[32988]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:39.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:38 smithi063 ceph-mon[28006]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:40.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.597+0000 7f9fce7fc700 1 -- 172.21.15.70:0/2179618454 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9fd0062e80 con 0x7f9fb80a3c40 2024-04-25T18:40:40.599 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 10 on host 'smithi070' 2024-04-25T18:40:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 msgr2=0x7f9fb80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 0x7f9fb80a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009550 tx=0x7f9fc80091b0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 msgr2=0x7f9fd0199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4006010 tx=0x7f9fc40047b0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 shutdown_connections 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9fb80a3c40 0x7f9fb80a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9fd01019a0 0x7f9fd0198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9fd0103540 0x7f9fd01a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 --2- 172.21.15.70:0/2179618454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fd0102ba0 0x7f9fd0199180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 >> 172.21.15.70:0/2179618454 conn(0x7f9fd00fcf40 msgr2=0x7f9fd0105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 shutdown_connections 2024-04-25T18:40:40.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-25T18:40:40.600+0000 7f9fd7834700 1 -- 172.21.15.70:0/2179618454 wait complete. 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:40.893 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:40 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:40 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:41.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:41.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:40 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:40:41.231 DEBUG:teuthology.orchestra.run.smithi070:osd.10> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T18:40:41.233 INFO:tasks.cephadm:Deploying osd.11 on smithi114 with /dev/vg_nvme/lv_1... 2024-04-25T18:40:41.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T18:40:41.479 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:40:41.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:41 smithi114 ceph-mon[32988]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T18:40:41.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:41 smithi114 ceph-mon[32988]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:41.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:41 smithi114 ceph-mon[32988]: osdmap e74: 11 total, 10 up, 11 in 2024-04-25T18:40:41.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:41 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:41.911 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:41 smithi114 ceph-mon[32988]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:41 smithi070 ceph-mon[32820]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T18:40:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:41 smithi070 ceph-mon[32820]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:41.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:41 smithi070 ceph-mon[32820]: osdmap e74: 11 total, 10 up, 11 in 2024-04-25T18:40:41.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:41 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:41.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:41 smithi070 ceph-mon[32820]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:42.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:41 smithi063 ceph-mon[28006]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T18:40:42.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:41 smithi063 ceph-mon[28006]: from='osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:42.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:41 smithi063 ceph-mon[28006]: osdmap e74: 11 total, 10 up, 11 in 2024-04-25T18:40:42.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:41 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:42.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:41 smithi063 ceph-mon[28006]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-04-25T18:40:42.694 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T18:40:42.421+0000 7f73a531c700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T18:40:42.694 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T18:40:42.435+0000 7f739d107700 -1 osd.10 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:40:42.865 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:42.865 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: osdmap e75: 11 total, 10 up, 11 in 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: from='osd.10 ' entity='osd.10' 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] boot 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T18:40:42.866 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:42 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:40:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: osdmap e75: 11 total, 10 up, 11 in 2024-04-25T18:40:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: from='osd.10 ' entity='osd.10' 2024-04-25T18:40:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] boot 2024-04-25T18:40:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T18:40:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:42 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-25T18:40:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-04-25T18:40:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: osdmap e75: 11 total, 10 up, 11 in 2024-04-25T18:40:43.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:43.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: from='osd.10 ' entity='osd.10' 2024-04-25T18:40:43.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: osd.10 [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] boot 2024-04-25T18:40:43.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: osdmap e76: 11 total, 11 up, 11 in 2024-04-25T18:40:43.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:42 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132221 s, 793 MB/s 2024-04-25T18:40:43.545 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2024-04-25T18:40:44.022 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:43 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:40:44.022 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:43 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:40:44.022 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:43 smithi070 ceph-mon[32820]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T18:40:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:43 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:40:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:43 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:40:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:43 smithi114 ceph-mon[32988]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T18:40:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:43 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:40:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:43 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:40:44.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:43 smithi063 ceph-mon[28006]: osdmap e77: 11 total, 11 up, 11 in 2024-04-25T18:40:44.240 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi114:vg_nvme/lv_1 2024-04-25T18:40:44.482 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:44.855 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:44.856 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:44 smithi114 ceph-mon[32988]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T18:40:45.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:44 smithi063 ceph-mon[28006]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T18:40:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:40:45.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:44 smithi070 ceph-mon[32820]: osdmap e78: 11 total, 11 up, 11 in 2024-04-25T18:40:45.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:45 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi070 2024-04-25T18:40:45.885 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:45 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 -- 172.21.15.114:0/2314251054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698103560 msgr2=0x7fb698109e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 --2- 172.21.15.114:0/2314251054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698103560 0x7fb698109e10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb688009990 tx=0x7fb688009ca0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 -- 172.21.15.114:0/2314251054 shutdown_connections 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 --2- 172.21.15.114:0/2314251054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698103560 0x7fb698109e10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 --2- 172.21.15.114:0/2314251054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6981019c0 0x7fb698101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 --2- 172.21.15.114:0/2314251054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698102bc0 0x7fb698103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 -- 172.21.15.114:0/2314251054 >> 172.21.15.114:0/2314251054 conn(0x7fb6980fcfa0 msgr2=0x7fb6980ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 -- 172.21.15.114:0/2314251054 shutdown_connections 2024-04-25T18:40:46.092 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.090+0000 7fb69e179700 1 -- 172.21.15.114:0/2314251054 wait complete. 2024-04-25T18:40:46.094 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 Processor -- start 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 -- start start 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6981019c0 0x7fb69806bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698103560 0x7fb6981a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb69810d550 con 0x7fb6981019c0 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb69810d3d0 con 0x7fb698103560 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb69e179700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb69810d6d0 con 0x7fb698102bc0 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb696ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb696ffd700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:56980/0 (socket says 172.21.15.114:56980) 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.091+0000 7fb696ffd700 1 -- 172.21.15.114:0/66677820 learned_addr learned my addr 172.21.15.114:0/66677820 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb697fff700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698103560 0x7fb6981a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 -- 172.21.15.114:0/66677820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698103560 msgr2=0x7fb6981a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:46.095 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698103560 0x7fb6981a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 -- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6981019c0 msgr2=0x7fb69806bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6981019c0 0x7fb69806bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 -- 172.21.15.114:0/66677820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6981a08e0 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb696ffd700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb68000b530 tx=0x7fb68000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680015020 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb680011ce0 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6981a0bd0 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680010d20 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.092+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6981a10c0 con 0x7fb698102bc0 2024-04-25T18:40:46.096 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.093+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb698069730 con 0x7fb698102bc0 2024-04-25T18:40:46.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.095+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb680018020 con 0x7fb698102bc0 2024-04-25T18:40:46.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.095+0000 7fb694ff9700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 0x7fb6840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:40:46.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.096+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6694+0+0 (secure 0 0 0) 0x7fb6800cfb80 con 0x7fb698102bc0 2024-04-25T18:40:46.097 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.096+0000 7fb6977fe700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 0x7fb6840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:40:46.098 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.097+0000 7fb6977fe700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 0x7fb6840a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb698102a20 tx=0x7fb690009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:40:46.101 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.099+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb68001f020 con 0x7fb698102bc0 2024-04-25T18:40:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:45 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi070 2024-04-25T18:40:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:45 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-25T18:40:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:45 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi070 2024-04-25T18:40:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:45 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 4272M 2024-04-25T18:40:46.307 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:40:46.305+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb698062e80 con 0x7fb6840a3c40 2024-04-25T18:40:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: pgmap v274: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T18:40:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: osdmap e79: 11 total, 11 up, 11 in 2024-04-25T18:40:47.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:47.062 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:47.062 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: pgmap v274: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T18:40:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: osdmap e79: 11 total, 11 up, 11 in 2024-04-25T18:40:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:47.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: pgmap v274: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T18:40:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: osdmap e79: 11 total, 11 up, 11 in 2024-04-25T18:40:47.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T18:40:47.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T18:40:47.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:48.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:47 smithi114 ceph-mon[32988]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:48.061 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:47 smithi114 ceph-mon[32988]: osdmap e80: 11 total, 11 up, 11 in 2024-04-25T18:40:48.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:47 smithi070 ceph-mon[32820]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:48.105 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:47 smithi070 ceph-mon[32820]: osdmap e80: 11 total, 11 up, 11 in 2024-04-25T18:40:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:47 smithi063 ceph-mon[28006]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:40:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:47 smithi063 ceph-mon[28006]: osdmap e80: 11 total, 11 up, 11 in 2024-04-25T18:40:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:48 smithi114 ceph-mon[32988]: pgmap v277: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:48 smithi063 ceph-mon[28006]: pgmap v277: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:48 smithi070 ceph-mon[32820]: pgmap v277: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 345 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:50.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/3279434757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]': finished 2024-04-25T18:40:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T18:40:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:40:50.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:49 smithi063 ceph-mon[28006]: from='client.? 172.21.15.114:0/3406582180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/3279434757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]': finished 2024-04-25T18:40:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T18:40:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:40:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:49 smithi070 ceph-mon[32820]: from='client.? 172.21.15.114:0/3406582180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:50.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/3279434757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]: dispatch 2024-04-25T18:40:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10cb8d84-790d-42b3-8f29-3771c44434e4"}]': finished 2024-04-25T18:40:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: osdmap e81: 12 total, 11 up, 12 in 2024-04-25T18:40:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:40:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:49 smithi114 ceph-mon[32988]: from='client.? 172.21.15.114:0/3406582180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T18:40:51.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:50 smithi114 ceph-mon[32988]: pgmap v279: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:51.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:50 smithi063 ceph-mon[28006]: pgmap v279: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:50 smithi070 ceph-mon[32820]: pgmap v279: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T18:40:52.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:51 smithi063 ceph-mon[28006]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T18:40:52.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:51 smithi063 ceph-mon[28006]: Cluster is now healthy 2024-04-25T18:40:52.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:51 smithi070 ceph-mon[32820]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T18:40:52.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:51 smithi070 ceph-mon[32820]: Cluster is now healthy 2024-04-25T18:40:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:51 smithi114 ceph-mon[32988]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T18:40:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:51 smithi114 ceph-mon[32988]: Cluster is now healthy 2024-04-25T18:40:53.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:52 smithi063 ceph-mon[28006]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:40:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:52 smithi070 ceph-mon[32820]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:40:53.289 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:52 smithi114 ceph-mon[32988]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T18:40:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:54 smithi063 ceph-mon[28006]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T18:40:55.171 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:54 smithi114 ceph-mon[32988]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T18:40:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:54 smithi070 ceph-mon[32820]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T18:40:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:56 smithi063 ceph-mon[28006]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T18:40:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T18:40:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:56 smithi070 ceph-mon[32820]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T18:40:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T18:40:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:57.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:56 smithi114 ceph-mon[32988]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T18:40:57.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T18:40:57.248 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:40:58.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:57 smithi070 ceph-mon[32820]: Deploying daemon osd.11 on smithi114 2024-04-25T18:40:58.272 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:57 smithi114 ceph-mon[32988]: Deploying daemon osd.11 on smithi114 2024-04-25T18:40:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:57 smithi063 ceph-mon[28006]: Deploying daemon osd.11 on smithi114 2024-04-25T18:40:59.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:40:58 smithi070 ceph-mon[32820]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T18:40:59.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:40:58 smithi114 ceph-mon[32988]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T18:40:59.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:40:58 smithi063 ceph-mon[28006]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T18:41:01.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:00 smithi070 ceph-mon[32820]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T18:41:01.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:00 smithi114 ceph-mon[32988]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T18:41:01.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:00 smithi063 ceph-mon[28006]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-25T18:41:03.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:02 smithi114 ceph-mon[32988]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T18:41:03.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.024 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.025 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:03.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:02 smithi070 ceph-mon[32820]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T18:41:03.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:03.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:02 smithi063 ceph-mon[28006]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T18:41:03.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:03.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:04 smithi114 ceph-mon[32988]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:05.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:04 smithi070 ceph-mon[32820]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:05.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:04 smithi063 ceph-mon[28006]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:05.892 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.891+0000 7fb694ff9700 1 -- 172.21.15.114:0/66677820 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fb698062e80 con 0x7fb6840a3c40 2024-04-25T18:41:05.892 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 11 on host 'smithi114' 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 msgr2=0x7fb6840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 0x7fb6840a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb698102a20 tx=0x7fb690009250 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 msgr2=0x7fb69819d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb68000b530 tx=0x7fb68000b8f0 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 shutdown_connections 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb6840a3c40 0x7fb6840a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb698103560 0x7fb6981a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6981019c0 0x7fb69806bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 --2- 172.21.15.114:0/66677820 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb698102bc0 0x7fb69819d9d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 >> 172.21.15.114:0/66677820 conn(0x7fb6980fcfa0 msgr2=0x7fb698105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 shutdown_connections 2024-04-25T18:41:05.895 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:05.893+0000 7fb69e179700 1 -- 172.21.15.114:0/66677820 wait complete. 2024-04-25T18:41:06.530 DEBUG:teuthology.orchestra.run.smithi114:osd.11> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T18:41:06.533 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T18:41:06.534 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.013 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:07.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:07.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:07.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:08.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:07 smithi114 ceph-mon[32988]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T18:41:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:07 smithi114 ceph-mon[32988]: osdmap e82: 12 total, 11 up, 12 in 2024-04-25T18:41:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:07 smithi114 ceph-mon[32988]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:07 smithi114 ceph-mon[32988]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.073 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:07 smithi063 ceph-mon[28006]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T18:41:08.073 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:07 smithi063 ceph-mon[28006]: osdmap e82: 12 total, 11 up, 12 in 2024-04-25T18:41:08.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:07 smithi063 ceph-mon[28006]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:08.074 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:07 smithi063 ceph-mon[28006]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:07 smithi070 ceph-mon[32820]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T18:41:08.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:07 smithi070 ceph-mon[32820]: osdmap e82: 12 total, 11 up, 12 in 2024-04-25T18:41:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:07 smithi070 ceph-mon[32820]: from='osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:07 smithi070 ceph-mon[32820]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2024-04-25T18:41:08.924 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T18:41:08.831+0000 7feb933dd700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: osdmap e83: 12 total, 11 up, 12 in 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.165 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:08 smithi063 ceph-mon[28006]: from='osd.11 ' entity='osd.11' 2024-04-25T18:41:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:41:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: osdmap e83: 12 total, 11 up, 12 in 2024-04-25T18:41:09.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:08 smithi070 ceph-mon[32820]: from='osd.11 ' entity='osd.11' 2024-04-25T18:41:09.317 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T18:41:08.973+0000 7feb8b1c8700 -1 osd.11 83 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T18:41:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2024-04-25T18:41:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: osdmap e83: 12 total, 11 up, 12 in 2024-04-25T18:41:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:09.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:08 smithi114 ceph-mon[32988]: from='osd.11 ' entity='osd.11' 2024-04-25T18:41:09.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 -- 172.21.15.63:0/2696641740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8105e60 msgr2=0x7f81a8108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:09.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 --2- 172.21.15.63:0/2696641740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8105e60 0x7f81a8108250 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8194009990 tx=0x7f8194009ca0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 -- 172.21.15.63:0/2696641740 shutdown_connections 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 --2- 172.21.15.63:0/2696641740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8108790 0x7f81a810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 --2- 172.21.15.63:0/2696641740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8103530 0x7f81a8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 --2- 172.21.15.63:0/2696641740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8105e60 0x7f81a8108250 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 -- 172.21.15.63:0/2696641740 >> 172.21.15.63:0/2696641740 conn(0x7f81a80fcf40 msgr2=0x7f81a80ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.433+0000 7f81ae600700 1 -- 172.21.15.63:0/2696641740 shutdown_connections 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.434+0000 7f81ae600700 1 -- 172.21.15.63:0/2696641740 wait complete. 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 Processor -- start 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 -- start start 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8105e60 0x7f81a819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8108790 0x7f81a81a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:09.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810e5a0 con 0x7f81a8108790 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810e420 con 0x7f81a8105e60 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.435+0000 7f81ae600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f81a810e720 con 0x7f81a8103530 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49926/0 (socket says 172.21.15.63:49926) 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 -- 172.21.15.63:0/3190337093 learned_addr learned my addr 172.21.15.63:0/3190337093 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7e00700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8108790 0x7f81a81a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 -- 172.21.15.63:0/3190337093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8105e60 msgr2=0x7f81a819b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8105e60 0x7f81a819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 -- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8108790 msgr2=0x7f81a81a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8108790 0x7f81a81a2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.436+0000 7f81a7400700 1 -- 172.21.15.63:0/3190337093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a81a2b20 con 0x7f81a8103530 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f81a7400700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f819800e9d0 tx=0x7f819800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819800cbd0 con 0x7f81a8103530 2024-04-25T18:41:09.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f819800cd70 con 0x7f81a8103530 2024-04-25T18:41:09.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a81a2e10 con 0x7f81a8103530 2024-04-25T18:41:09.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198018680 con 0x7f81a8103530 2024-04-25T18:41:09.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.437+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a81a32c0 con 0x7f81a8103530 2024-04-25T18:41:09.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.438+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a8069730 con 0x7f81a8103530 2024-04-25T18:41:09.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.440+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8198010430 con 0x7f81a8103530 2024-04-25T18:41:09.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.441+0000 7f819fe00700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 0x7f818c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:09.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.442+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 6847+0+0 (secure 0 0 0) 0x7f81980d00b0 con 0x7f81a8103530 2024-04-25T18:41:09.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.442+0000 7f81a6a00700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 0x7f818c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:09.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.442+0000 7f81a6a00700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 0x7f818c0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81940098e0 tx=0x7f819400b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:09.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.445+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f819809cfb0 con 0x7f81a8103530 2024-04-25T18:41:09.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.682+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f81a8063680 con 0x7f81a8103530 2024-04-25T18:41:09.684 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:09.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.682+0000 7f819fe00700 1 -- 172.21.15.63:0/3190337093 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f819809c8b0 con 0x7f81a8103530 2024-04-25T18:41:09.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 msgr2=0x7f818c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:09.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 0x7f818c0a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f81940098e0 tx=0x7f819400b610 comp rx=0 tx=0).stop 2024-04-25T18:41:09.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 msgr2=0x7f81a819aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:09.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f819800e9d0 tx=0x7f819800ece0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 shutdown_connections 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f818c0a3cc0 0x7f818c0a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81a8103530 0x7f81a819aec0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a8108790 0x7f81a81a2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 --2- 172.21.15.63:0/3190337093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f81a8105e60 0x7f81a819b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.686+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 >> 172.21.15.63:0/3190337093 conn(0x7f81a80fcf40 msgr2=0x7f81a8102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.687+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 shutdown_connections 2024-04-25T18:41:09.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:09.687+0000 7f81ae600700 1 -- 172.21.15.63:0/3190337093 wait complete. 2024-04-25T18:41:10.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:09 smithi114 ceph-mon[32988]: purged_snaps scrub starts 2024-04-25T18:41:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:09 smithi114 ceph-mon[32988]: purged_snaps scrub ok 2024-04-25T18:41:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:09 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3190337093' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:09 smithi063 ceph-mon[28006]: purged_snaps scrub starts 2024-04-25T18:41:10.123 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:09 smithi063 ceph-mon[28006]: purged_snaps scrub ok 2024-04-25T18:41:10.124 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.124 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:09 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3190337093' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:10.124 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:09 smithi070 ceph-mon[32820]: purged_snaps scrub starts 2024-04-25T18:41:10.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:09 smithi070 ceph-mon[32820]: purged_snaps scrub ok 2024-04-25T18:41:10.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:09 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3190337093' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:10.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:10.546 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":83,"num_osds":12,"num_up_osds":11,"osd_up_since":1714070442,"num_in_osds":12,"osd_in_since":1714070449,"num_remapped_pgs":0} 2024-04-25T18:41:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] boot 2024-04-25T18:41:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T18:41:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] boot 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:11.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 983 GiB / 983 GiB avail 2024-04-25T18:41:11.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: osd.11 [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] boot 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:11.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T18:41:12.012 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:12.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: Detected new or changed devices on smithi114 2024-04-25T18:41:12.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-25T18:41:12.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T18:41:12.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:12.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:12.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:11 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:12.228 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: Detected new or changed devices on smithi114 2024-04-25T18:41:12.228 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-25T18:41:12.228 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T18:41:12.229 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:12.229 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:12.229 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:11 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: Detected new or changed devices on smithi114 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 4272M 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:11 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:13.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:12 smithi070 ceph-mon[32820]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:13.242 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:12 smithi063 ceph-mon[28006]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:13.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:12 smithi114 ceph-mon[32988]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:14.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.327+0000 7fae7e000700 1 -- 172.21.15.63:0/2955512948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 msgr2=0x7fae780758c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:14.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.327+0000 7fae7e000700 1 --2- 172.21.15.63:0/2955512948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae780758c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fae60009990 tx=0x7fae60009ca0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 -- 172.21.15.63:0/2955512948 shutdown_connections 2024-04-25T18:41:14.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 --2- 172.21.15.63:0/2955512948 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae7810a9d0 0x7fae7810ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 --2- 172.21.15.63:0/2955512948 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae78076a10 0x7fae78074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 --2- 172.21.15.63:0/2955512948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae780758c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 -- 172.21.15.63:0/2955512948 >> 172.21.15.63:0/2955512948 conn(0x7fae780fe090 msgr2=0x7fae781004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:14.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 -- 172.21.15.63:0/2955512948 shutdown_connections 2024-04-25T18:41:14.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.328+0000 7fae7e000700 1 -- 172.21.15.63:0/2955512948 wait complete. 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 Processor -- start 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 -- start start 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae78076a10 0x7fae7819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 0x7fae781a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fae78110770 con 0x7fae78075440 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae781105f0 con 0x7fae7810a9d0 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae7e000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fae781108f0 con 0x7fae78076a10 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.329+0000 7fae77400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:14.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51496/0 (socket says 172.21.15.63:51496) 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 -- 172.21.15.63:0/1651585154 learned_addr learned my addr 172.21.15.63:0/1651585154 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae6ea00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae78076a10 0x7fae7819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77e00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 0x7fae781a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 -- 172.21.15.63:0/1651585154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae78076a10 msgr2=0x7fae7819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae78076a10 0x7fae7819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 -- 172.21.15.63:0/1651585154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 msgr2=0x7fae781a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 0x7fae781a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 -- 172.21.15.63:0/1651585154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae781a2a00 con 0x7fae78075440 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77e00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 0x7fae781a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.330+0000 7fae77400700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fae6800e9a0 tx=0x7fae6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.331+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6800cbd0 con 0x7fae78075440 2024-04-25T18:41:14.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.331+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae6800cd70 con 0x7fae78075440 2024-04-25T18:41:14.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.331+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae781a2cf0 con 0x7fae78075440 2024-04-25T18:41:14.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.331+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68018680 con 0x7fae78075440 2024-04-25T18:41:14.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.331+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae781a3180 con 0x7fae78075440 2024-04-25T18:41:14.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.332+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae78069730 con 0x7fae78075440 2024-04-25T18:41:14.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.334+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fae68010430 con 0x7fae78075440 2024-04-25T18:41:14.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.336+0000 7fae74c00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 0x7fae5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:14.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.336+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fae680d0220 con 0x7fae78075440 2024-04-25T18:41:14.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.336+0000 7fae6ea00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 0x7fae5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:14.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.337+0000 7fae6ea00700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 0x7fae5c0a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae6000b2e0 tx=0x7fae60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:14.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.341+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fae6809d000 con 0x7fae78075440 2024-04-25T18:41:14.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.575+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fae78063650 con 0x7fae78075440 2024-04-25T18:41:14.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.577+0000 7fae74c00700 1 -- 172.21.15.63:0/1651585154 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v85) ==== 74+0+133 (secure 0 0 0) 0x7fae6809c900 con 0x7fae78075440 2024-04-25T18:41:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:14.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 msgr2=0x7fae5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:14.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 0x7fae5c0a6100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fae6000b2e0 tx=0x7fae60005fb0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 msgr2=0x7fae7819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:14.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fae6800e9a0 tx=0x7fae6800ecb0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 shutdown_connections 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fae5c0a3c40 0x7fae5c0a6100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7810a9d0 0x7fae781a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae78075440 0x7fae7819adb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 --2- 172.21.15.63:0/1651585154 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fae78076a10 0x7fae7819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 >> 172.21.15.63:0/1651585154 conn(0x7fae780fe090 msgr2=0x7fae78106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 shutdown_connections 2024-04-25T18:41:14.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:14.580+0000 7fae7e000700 1 -- 172.21.15.63:0/1651585154 wait complete. 2024-04-25T18:41:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:14 smithi070 ceph-mon[32820]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:14 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1651585154' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:15.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:14 smithi114 ceph-mon[32988]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:15.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:14 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1651585154' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:15.366 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:14 smithi063 ceph-mon[28006]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:15.367 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:14 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1651585154' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T18:41:15.442 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":85,"num_osds":12,"num_up_osds":12,"osd_up_since":1714070469,"num_in_osds":12,"osd_in_since":1714070449,"num_remapped_pgs":0} 2024-04-25T18:41:15.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:41:15.898 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:17.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:16 smithi114 ceph-mon[32988]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:17.341 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:16 smithi063 ceph-mon[28006]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:17.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:16 smithi070 ceph-mon[32820]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:18.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.278+0000 7f987d800700 1 -- 172.21.15.63:0/962281146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 msgr2=0x7f9878111620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:18.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.278+0000 7f987d800700 1 --2- 172.21.15.63:0/962281146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f9878111620 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f987000b0a0 tx=0x7f987000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 -- 172.21.15.63:0/962281146 shutdown_connections 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 --2- 172.21.15.63:0/962281146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 0x7f98780751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 --2- 172.21.15.63:0/962281146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f9878111620 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 --2- 172.21.15.63:0/962281146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987810ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 -- 172.21.15.63:0/962281146 >> 172.21.15.63:0/962281146 conn(0x7f98780fe4c0 msgr2=0x7f98781008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 -- 172.21.15.63:0/962281146 shutdown_connections 2024-04-25T18:41:18.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.279+0000 7f987d800700 1 -- 172.21.15.63:0/962281146 wait complete. 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.280+0000 7f987d800700 1 Processor -- start 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 -- start start 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 0x7f987819b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f987819fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9878114d90 con 0x7f987810f210 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9878114c10 con 0x7f9878075730 2024-04-25T18:41:18.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987d800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9878114f10 con 0x7f9878074e10 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f987ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f987819fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f986fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f986fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36262/0 (socket says 172.21.15.63:36262) 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f9877e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 0x7f987819b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.281+0000 7f986fe00700 1 -- 172.21.15.63:0/1309815752 learned_addr learned my addr 172.21.15.63:0/1309815752 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 -- 172.21.15.63:0/1309815752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 msgr2=0x7f987819b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 0x7f987819b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 -- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 msgr2=0x7f987819fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f987819fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 -- 172.21.15.63:0/1309815752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98781a0220 con 0x7f9878075730 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f986fe00700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f986000e9a0 tx=0x7f986000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986000ca10 con 0x7f9878075730 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986000cbb0 con 0x7f9878075730 2024-04-25T18:41:18.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.282+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98781a0510 con 0x7f9878075730 2024-04-25T18:41:18.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.283+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860018680 con 0x7f9878075730 2024-04-25T18:41:18.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.283+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98781a7d90 con 0x7f9878075730 2024-04-25T18:41:18.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.284+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9878069730 con 0x7f9878075730 2024-04-25T18:41:18.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.286+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9860010430 con 0x7f9878075730 2024-04-25T18:41:18.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.287+0000 7f9875600700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 0x7f985c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:18.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.287+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f98600d00e0 con 0x7f9878075730 2024-04-25T18:41:18.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.287+0000 7f9877e00700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 0x7f985c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:18.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.288+0000 7f9877e00700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 0x7f985c0a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9868009c60 tx=0x7f9868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:18.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.291+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f986009cec0 con 0x7f9878075730 2024-04-25T18:41:18.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.522+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f987802df80 con 0x7f9878075730 2024-04-25T18:41:18.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.523+0000 7f9875600700 1 -- 172.21.15.63:0/1309815752 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+18409 (secure 0 0 0) 0x7f986009c7c0 con 0x7f9878075730 2024-04-25T18:41:18.525 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:18.525 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":85,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:41:11.180016+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:41:18.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 msgr2=0x7f985c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:18.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 0x7f985c0a6100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9868009c60 tx=0x7f9868009260 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 msgr2=0x7f987819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f986000e9a0 tx=0x7f986000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 shutdown_connections 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f985c0a3c40 0x7f985c0a6100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9878074e10 0x7f987819b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f987810f210 0x7f987819fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 --2- 172.21.15.63:0/1309815752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9878075730 0x7f987819b7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 >> 172.21.15.63:0/1309815752 conn(0x7f98780fe4c0 msgr2=0x7f98780ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.526+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 shutdown_connections 2024-04-25T18:41:18.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:18.527+0000 7f987d800700 1 -- 172.21.15.63:0/1309815752 wait complete. 2024-04-25T18:41:19.255 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:18 smithi070 ceph-mon[32820]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:19.255 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:18 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1309815752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:41:19.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:18 smithi063 ceph-mon[28006]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:19.301 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:18 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1309815752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:41:19.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:18 smithi114 ceph-mon[32988]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:19.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:18 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1309815752' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:41:19.380 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T18:36:47.142075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T18:41:19.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T18:41:19.849 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:21.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:20 smithi114 ceph-mon[32988]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:21.197 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:20 smithi063 ceph-mon[28006]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:21.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:20 smithi070 ceph-mon[32820]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:22.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 -- 172.21.15.63:0/3103532439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3010f440 msgr2=0x7f3b30111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 --2- 172.21.15.63:0/3103532439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3010f440 0x7f3b30111830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009f90 tx=0x7f3b2800b330 comp rx=0 tx=0).stop 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 -- 172.21.15.63:0/3103532439 shutdown_connections 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 --2- 172.21.15.63:0/3103532439 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b30074c70 0x7f3b30075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 --2- 172.21.15.63:0/3103532439 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3010f440 0x7f3b30111830 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 --2- 172.21.15.63:0/3103532439 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30075590 0x7f3b3010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 -- 172.21.15.63:0/3103532439 >> 172.21.15.63:0/3103532439 conn(0x7f3b30079ba0 msgr2=0x7f3b30079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 -- 172.21.15.63:0/3103532439 shutdown_connections 2024-04-25T18:41:22.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.216+0000 7f3b34a00700 1 -- 172.21.15.63:0/3103532439 wait complete. 2024-04-25T18:41:22.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.217+0000 7f3b34a00700 1 Processor -- start 2024-04-25T18:41:22.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 -- start start 2024-04-25T18:41:22.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 0x7f3b3019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 0x7f3b3019fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30114e20 con 0x7f3b30074c70 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30114ca0 con 0x7f3b30075590 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b34a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3b30114fa0 con 0x7f3b3010f440 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b2de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b2de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42768/0 (socket says 172.21.15.63:42768) 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b2e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 0x7f3b3019fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.218+0000 7f3b2de00700 1 -- 172.21.15.63:0/2103890539 learned_addr learned my addr 172.21.15.63:0/2103890539 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:22.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b25400700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 0x7f3b3019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 -- 172.21.15.63:0/2103890539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 msgr2=0x7f3b3019fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 0x7f3b3019fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 -- 172.21.15.63:0/2103890539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 msgr2=0x7f3b3019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 0x7f3b3019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 -- 172.21.15.63:0/2103890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b301a0270 con 0x7f3b30074c70 2024-04-25T18:41:22.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b25400700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 0x7f3b3019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2e800700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 0x7f3b3019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.219+0000 7f3b2de00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9a0 tx=0x7f3b2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.220+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000ca10 con 0x7f3b30074c70 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.220+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b301a0560 con 0x7f3b30074c70 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.220+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b2000cbb0 con 0x7f3b30074c70 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.220+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b20018680 con 0x7f3b30074c70 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.220+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301a7de0 con 0x7f3b30074c70 2024-04-25T18:41:22.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.222+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b30069730 con 0x7f3b30074c70 2024-04-25T18:41:22.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.224+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3b20010430 con 0x7f3b30074c70 2024-04-25T18:41:22.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.225+0000 7f3b27400700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 0x7f3b100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:22.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.225+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f3b200d0130 con 0x7f3b30074c70 2024-04-25T18:41:22.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.225+0000 7f3b25400700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 0x7f3b100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:22.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.226+0000 7f3b25400700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 0x7f3b100a6180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009750 tx=0x7f3b18009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:22.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.230+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b2009cf10 con 0x7f3b30074c70 2024-04-25T18:41:22.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.460+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b3002df80 con 0x7f3b30074c70 2024-04-25T18:41:22.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.461+0000 7f3b27400700 1 -- 172.21.15.63:0/2103890539 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f3b2009c810 con 0x7f3b30074c70 2024-04-25T18:41:22.463 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 msgr2=0x7f3b100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 0x7f3b100a6180 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009750 tx=0x7f3b18009310 comp rx=0 tx=0).stop 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 msgr2=0x7f3b3019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e9a0 tx=0x7f3b2000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 shutdown_connections 2024-04-25T18:41:22.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3b100a3cc0 0x7f3b100a6180 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.464+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3b3010f440 0x7f3b3019fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.465+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b30074c70 0x7f3b3019b2b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.465+0000 7f3b34a00700 1 --2- 172.21.15.63:0/2103890539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b30075590 0x7f3b3019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.465+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 >> 172.21.15.63:0/2103890539 conn(0x7f3b30079ba0 msgr2=0x7f3b300ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.465+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 shutdown_connections 2024-04-25T18:41:22.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:22.465+0000 7f3b34a00700 1 -- 172.21.15.63:0/2103890539 wait complete. 2024-04-25T18:41:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:22 smithi063 ceph-mon[28006]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:22 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2103890539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T18:41:23.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:22 smithi114 ceph-mon[32988]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:23.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:22 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2103890539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.a on smithi063 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.d on smithi063 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.g on smithi063 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.j on smithi063 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.b on smithi070 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.e on smithi070 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.h on smithi070 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.k on smithi070 2024-04-25T18:41:23.335 INFO:tasks.cephadm:Adding mds.c on smithi114 2024-04-25T18:41:23.336 INFO:tasks.cephadm:Adding mds.f on smithi114 2024-04-25T18:41:23.336 INFO:tasks.cephadm:Adding mds.i on smithi114 2024-04-25T18:41:23.336 INFO:tasks.cephadm:Adding mds.l on smithi114 2024-04-25T18:41:23.336 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l' 2024-04-25T18:41:23.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:22 smithi070 ceph-mon[32820]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:23.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:22 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2103890539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T18:41:23.582 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T18:41:24.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:23 smithi114 ceph-mon[32988]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:23 smithi063 ceph-mon[28006]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:23 smithi070 ceph-mon[32820]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 -- 172.21.15.114:0/2369628370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414075440 msgr2=0x7f14140758c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 --2- 172.21.15.114:0/2369628370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414075440 0x7f14140758c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009990 tx=0x7f13fc009ca0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 -- 172.21.15.114:0/2369628370 shutdown_connections 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 --2- 172.21.15.114:0/2369628370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f141410a980 0x7f141410ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 --2- 172.21.15.114:0/2369628370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414076a10 0x7f1414074e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 --2- 172.21.15.114:0/2369628370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1414075440 0x7f14140758c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 -- 172.21.15.114:0/2369628370 >> 172.21.15.114:0/2369628370 conn(0x7f14140fe040 msgr2=0x7f1414100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 -- 172.21.15.114:0/2369628370 shutdown_connections 2024-04-25T18:41:25.253 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.248+0000 7f1419852700 1 -- 172.21.15.114:0/2369628370 wait complete. 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 Processor -- start 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 -- start start 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1414075440 0x7f141419aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414076a10 0x7f141419b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f14141107f0 con 0x7f1414076a10 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1414110670 con 0x7f1414075440 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1419852700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1414110970 con 0x7f141410a980 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f14137fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f14137fe700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.114:43342/0 (socket says 172.21.15.114:43342) 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f14137fe700 1 -- 172.21.15.114:0/4122438991 learned_addr learned my addr 172.21.15.114:0/4122438991 (peer_addr_for_me v2:172.21.15.114:0/0) 2024-04-25T18:41:25.254 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.249+0000 7f1412ffd700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1414075440 0x7f141419aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 -- 172.21.15.114:0/4122438991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1414075440 msgr2=0x7f141419aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1414075440 0x7f141419aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 -- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414076a10 msgr2=0x7f141419b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414076a10 0x7f141419b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 -- 172.21.15.114:0/4122438991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14141a2b40 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f14137fe700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b9a0 tx=0x7f140c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c019070 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14141a2e30 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f140c014440 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c00ce60 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.250+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141a3360 con 0x7f141410a980 2024-04-25T18:41:25.255 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.251+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1414069730 con 0x7f141410a980 2024-04-25T18:41:25.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.252+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f140c01c070 con 0x7f141410a980 2024-04-25T18:41:25.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.253+0000 7f1418850700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 0x7f14000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:25.256 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.253+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f140c0cf320 con 0x7f141410a980 2024-04-25T18:41:25.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.253+0000 7f1412ffd700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 0x7f14000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:25.257 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.254+0000 7f1412ffd700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 0x7f14000a6100 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1404006fd0 tx=0x7f1404008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:25.259 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.257+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f140c09c230 con 0x7f141410a980 2024-04-25T18:41:25.470 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.468+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l", "target": ["mon-mgr", ""]}) -- 0x7f1414063080 con 0x7f14000a3c40 2024-04-25T18:41:25.488 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.486+0000 7f1418850700 1 -- 172.21.15.114:0/4122438991 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f1414063080 con 0x7f14000a3c40 2024-04-25T18:41:25.488 INFO:teuthology.orchestra.run.smithi114.stdout:Scheduled mds.all update... 2024-04-25T18:41:25.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 msgr2=0x7f14000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:25.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 0x7f14000a6100 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1404006fd0 tx=0x7f1404008040 comp rx=0 tx=0).stop 2024-04-25T18:41:25.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 msgr2=0x7f14141a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:25.490 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b9a0 tx=0x7f140c00bd60 comp rx=0 tx=0).stop 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 shutdown_connections 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14000a3c40 0x7f14000a6100 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f141410a980 0x7f14141a24a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1414075440 0x7f141419aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 --2- 172.21.15.114:0/4122438991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1414076a10 0x7f141419b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.488+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 >> 172.21.15.114:0/4122438991 conn(0x7f14140fe040 msgr2=0x7f14141068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.489+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 shutdown_connections 2024-04-25T18:41:25.491 INFO:teuthology.orchestra.run.smithi114.stderr:2024-04-25T18:41:25.489+0000 7f1419852700 1 -- 172.21.15.114:0/4122438991 wait complete. 2024-04-25T18:41:26.151 DEBUG:teuthology.orchestra.run.smithi063:mds.a> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T18:41:26.159 DEBUG:teuthology.orchestra.run.smithi063:mds.d> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T18:41:26.204 DEBUG:teuthology.orchestra.run.smithi063:mds.g> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T18:41:26.251 DEBUG:teuthology.orchestra.run.smithi063:mds.j> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T18:41:26.297 DEBUG:teuthology.orchestra.run.smithi070:mds.b> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T18:41:26.300 DEBUG:teuthology.orchestra.run.smithi070:mds.e> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T18:41:26.343 DEBUG:teuthology.orchestra.run.smithi070:mds.h> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T18:41:26.386 DEBUG:teuthology.orchestra.run.smithi070:mds.k> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T18:41:26.429 DEBUG:teuthology.orchestra.run.smithi114:mds.c> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T18:41:26.431 DEBUG:teuthology.orchestra.run.smithi114:mds.f> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T18:41:26.474 DEBUG:teuthology.orchestra.run.smithi114:mds.i> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T18:41:26.517 DEBUG:teuthology.orchestra.run.smithi114:mds.l> sudo journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T18:41:26.562 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T18:41:26.562 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:26 smithi070 ceph-mon[32820]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:26 smithi070 ceph-mon[32820]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:41:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:26 smithi070 ceph-mon[32820]: Saving service mds.all spec with placement smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l;count:12 2024-04-25T18:41:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:26.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:26 smithi114 ceph-mon[32988]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:26 smithi114 ceph-mon[32988]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:41:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:26 smithi114 ceph-mon[32988]: Saving service mds.all spec with placement smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l;count:12 2024-04-25T18:41:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:26 smithi063 ceph-mon[28006]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:26 smithi063 ceph-mon[28006]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:41:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:26 smithi063 ceph-mon[28006]: Saving service mds.all spec with placement smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l;count:12 2024-04-25T18:41:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:41:27.021 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.677 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:27 smithi114 ceph-mon[32988]: Deploying daemon mds.l on smithi114 2024-04-25T18:41:27.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:27.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:27.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:27.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:27.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.770 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:27 smithi063 ceph-mon[28006]: Deploying daemon mds.l on smithi114 2024-04-25T18:41:27.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:27.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:27 smithi070 ceph-mon[32820]: Deploying daemon mds.l on smithi114 2024-04-25T18:41:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:28 smithi063 ceph-mon[28006]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:28 smithi114 ceph-mon[32988]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:29.067 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:28 smithi114 systemd[1]: Starting Ceph mds.l for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:28 smithi070 ceph-mon[32820]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 podman[62764]: 2024-04-25 18:41:29.08233667 +0000 UTC m=+0.113250888 container create 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 podman[62764]: 2024-04-25 18:41:28.987589684 +0000 UTC m=+0.018503917 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 podman[62764]: 2024-04-25 18:41:29.292842859 +0000 UTC m=+0.323757083 container init 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 podman[62764]: 2024-04-25 18:41:29.295434656 +0000 UTC m=+0.326348880 container start 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 ceph-mds[62783]: -- 172.21.15.114:0/3219135266 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563002459a40 con 0x56300242cc00 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l[62779]: starting mds.l at 2024-04-25T18:41:29.394 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 bash[62764]: 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 -- 172.21.15.63:0/3701182543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 msgr2=0x7f69c8101420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 --2- 172.21.15.63:0/3701182543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c8101420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009990 tx=0x7f69b8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 -- 172.21.15.63:0/3701182543 shutdown_connections 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 --2- 172.21.15.63:0/3701182543 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c8101420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 --2- 172.21.15.63:0/3701182543 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c810f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 --2- 172.21.15.63:0/3701182543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 0x7f69c810c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.509+0000 7f69c7e00700 1 -- 172.21.15.63:0/3701182543 >> 172.21.15.63:0/3701182543 conn(0x7f69c80766e0 msgr2=0x7f69c8076af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:29.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.510+0000 7f69c7e00700 1 -- 172.21.15.63:0/3701182543 shutdown_connections 2024-04-25T18:41:29.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.510+0000 7f69c7e00700 1 -- 172.21.15.63:0/3701182543 wait complete. 2024-04-25T18:41:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 Processor -- start 2024-04-25T18:41:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 -- start start 2024-04-25T18:41:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 0x7f69c819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69c8112bc0 con 0x7f69c8101960 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69c8112a40 con 0x7f69c810cfc0 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.511+0000 7f69c7e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69c8112d40 con 0x7f69c8101040 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56164/0 (socket says 172.21.15.63:56164) 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 -- 172.21.15.63:0/2899305227 learned_addr learned my addr 172.21.15.63:0/2899305227 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6000700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 0x7f69c819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c7400700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 -- 172.21.15.63:0/2899305227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 msgr2=0x7f69c819fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c819fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 -- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 msgr2=0x7f69c819b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 0x7f69c819b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 -- 172.21.15.63:0/2899305227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c81a02f0 con 0x7f69c8101040 2024-04-25T18:41:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c7400700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:41:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.512+0000 7f69c6a00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69b8000c00 tx=0x7f69b8009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.513+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8027070 con 0x7f69c8101040 2024-04-25T18:41:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.513+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69b8022470 con 0x7f69c8101040 2024-04-25T18:41:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.513+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8017650 con 0x7f69c8101040 2024-04-25T18:41:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.513+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c81a0580 con 0x7f69c8101040 2024-04-25T18:41:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.513+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c81a7e60 con 0x7f69c8101040 2024-04-25T18:41:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.514+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c8069730 con 0x7f69c8101040 2024-04-25T18:41:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.516+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69b802a070 con 0x7f69c8101040 2024-04-25T18:41:29.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.518+0000 7f69bf400700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 0x7f69ac0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.518+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f69b80d0f00 con 0x7f69c8101040 2024-04-25T18:41:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.518+0000 7f69c6000700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 0x7f69ac0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.519+0000 7f69c6000700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 0x7f69ac0a6180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69c810eb60 tx=0x7f69b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:29.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.522+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69b805eff0 con 0x7f69c8101040 2024-04-25T18:41:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.810+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69c81a8030 con 0x7f69c8101040 2024-04-25T18:41:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.811+0000 7f69bf400700 1 -- 172.21.15.63:0/2899305227 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f69b8013070 con 0x7f69c8101040 2024-04-25T18:41:29.812 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:29.812 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T18:41:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 msgr2=0x7f69ac0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 0x7f69ac0a6180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69c810eb60 tx=0x7f69b4009260 comp rx=0 tx=0).stop 2024-04-25T18:41:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 msgr2=0x7f69c819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f69b8000c00 tx=0x7f69b8009510 comp rx=0 tx=0).stop 2024-04-25T18:41:29.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 shutdown_connections 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69ac0a3cc0 0x7f69ac0a6180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69c8101040 0x7f69c819b330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69c810cfc0 0x7f69c819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 --2- 172.21.15.63:0/2899305227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69c8101960 0x7f69c819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 >> 172.21.15.63:0/2899305227 conn(0x7f69c80766e0 msgr2=0x7f69c8102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 shutdown_connections 2024-04-25T18:41:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:29.814+0000 7f69c7e00700 1 -- 172.21.15.63:0/2899305227 wait complete. 2024-04-25T18:41:29.816 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 18:41:29 smithi114 systemd[1]: Started Ceph mds.l for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 2 2024-04-25T18:41:30.400 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6832/1215662020,v1:172.21.15.114:6833/1215662020] up:boot 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: fsmap 1 up:standby 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.401 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.402 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:30.402 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:30.402 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:30.402 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: Deploying daemon mds.k on smithi070 2024-04-25T18:41:30.402 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:30 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2899305227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:30.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:30.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6832/1215662020,v1:172.21.15.114:6833/1215662020] up:boot 2024-04-25T18:41:30.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: fsmap 1 up:standby 2024-04-25T18:41:30.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:41:30.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: Deploying daemon mds.k on smithi070 2024-04-25T18:41:30.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:30 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2899305227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6832/1215662020,v1:172.21.15.114:6833/1215662020] up:boot 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: fsmap 1 up:standby 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:30.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: Deploying daemon mds.k on smithi070 2024-04-25T18:41:30.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:30 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2899305227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:31.703 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:31 smithi070 systemd[1]: Starting Ceph mds.k for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:31 smithi070 podman[62570]: 2024-04-25 18:41:31.807362788 +0000 UTC m=+0.113079821 container create 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:31 smithi070 podman[62570]: 2024-04-25 18:41:31.712586984 +0000 UTC m=+0.018304031 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 podman[62570]: 2024-04-25 18:41:32.026285146 +0000 UTC m=+0.332002177 container init 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 podman[62570]: 2024-04-25 18:41:32.028805378 +0000 UTC m=+0.334522421 container start 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mds[62588]: -- 172.21.15.70:0/545394665 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5652f5f37a40 con 0x5652f5f0ac00 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k[62584]: starting mds.k at 2024-04-25T18:41:32.119 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 bash[62570]: 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 2024-04-25T18:41:32.453 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 18:41:32 smithi070 systemd[1]: Started Ceph mds.k for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:32.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:32 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:32 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:32.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:32.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:32 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:33.114 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:33.444 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:33 smithi070 ceph-mon[32820]: Deploying daemon mds.j on smithi063 2024-04-25T18:41:33.444 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:33 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6832/2179103180,v1:172.21.15.70:6833/2179103180] up:boot 2024-04-25T18:41:33.444 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:33 smithi070 ceph-mon[32820]: fsmap 2 up:standby 2024-04-25T18:41:33.445 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:33 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:41:33.511 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:33 smithi063 ceph-mon[28006]: Deploying daemon mds.j on smithi063 2024-04-25T18:41:33.511 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:33 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6832/2179103180,v1:172.21.15.70:6833/2179103180] up:boot 2024-04-25T18:41:33.511 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:33 smithi063 ceph-mon[28006]: fsmap 2 up:standby 2024-04-25T18:41:33.511 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:33 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:41:33.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:33 smithi114 ceph-mon[32988]: Deploying daemon mds.j on smithi063 2024-04-25T18:41:33.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:33 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6832/2179103180,v1:172.21.15.70:6833/2179103180] up:boot 2024-04-25T18:41:33.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:33 smithi114 ceph-mon[32988]: fsmap 2 up:standby 2024-04-25T18:41:33.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:33 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:41:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:34 smithi063 ceph-mon[28006]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:34.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:34 smithi070 ceph-mon[32820]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:34.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:34 smithi114 ceph-mon[32988]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:36.024 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:35 smithi063 systemd[1]: Starting Ceph mds.j for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:36.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 -- 172.21.15.63:0/358816474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 msgr2=0x7f092810f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:36.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 --2- 172.21.15.63:0/358816474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092810f8b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0920009f90 tx=0x7f092000b330 comp rx=0 tx=0).stop 2024-04-25T18:41:36.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 -- 172.21.15.63:0/358816474 shutdown_connections 2024-04-25T18:41:36.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 --2- 172.21.15.63:0/358816474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092810f8b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 --2- 172.21.15.63:0/358816474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f0928108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 --2- 172.21.15.63:0/358816474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f0928102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.597+0000 7f092dc00700 1 -- 172.21.15.63:0/358816474 >> 172.21.15.63:0/358816474 conn(0x7f0928079ba0 msgr2=0x7f0928079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:36.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.598+0000 7f092dc00700 1 -- 172.21.15.63:0/358816474 shutdown_connections 2024-04-25T18:41:36.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.598+0000 7f092dc00700 1 -- 172.21.15.63:0/358816474 wait complete. 2024-04-25T18:41:36.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.599+0000 7f092dc00700 1 Processor -- start 2024-04-25T18:41:36.601 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:36 smithi063 ceph-mon[28006]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.599+0000 7f092dc00700 1 -- start start 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.599+0000 7f092dc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f092819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.599+0000 7f092dc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f092dc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f092819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f092dc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112c50 con 0x7f0928102a90 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f092dc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112ad0 con 0x7f0928108a50 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f092dc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0928112dd0 con 0x7f0928103450 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f092819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56180/0 (socket says 172.21.15.63:56180) 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 -- 172.21.15.63:0/287756841 learned_addr learned my addr 172.21.15.63:0/287756841 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:36.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0927400700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f092819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 -- 172.21.15.63:0/287756841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 msgr2=0x7f092819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f092819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 -- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 msgr2=0x7f092819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f092819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926000700 1 -- 172.21.15.63:0/287756841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09281a0350 con 0x7f0928103450 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0926a00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f092819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.600+0000 7f0927400700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f092819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f0926000700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f091400ec10 tx=0x7f091400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091400cc90 con 0x7f0928103450 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f091400ce30 con 0x7f0928103450 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09281a0640 con 0x7f0928103450 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914010430 con 0x7f0928103450 2024-04-25T18:41:36.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.601+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a7ec0 con 0x7f0928103450 2024-04-25T18:41:36.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.602+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0928069730 con 0x7f0928103450 2024-04-25T18:41:36.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.605+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0914004020 con 0x7f0928103450 2024-04-25T18:41:36.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.606+0000 7f091f400700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 0x7f090c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:36.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.606+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f09140d0200 con 0x7f0928103450 2024-04-25T18:41:36.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.606+0000 7f0926a00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 0x7f090c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:36.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.607+0000 7f0926a00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 0x7f090c0a6180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fd0 tx=0x7f0918009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:36.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.610+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f091409cfe0 con 0x7f0928103450 2024-04-25T18:41:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:36 smithi070 ceph-mon[32820]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:36.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:36 smithi114 ceph-mon[32988]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:36.878 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:36 smithi063 podman[68683]: 2024-04-25 18:41:36.591430723 +0000 UTC m=+0.035941604 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:36.878 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:36 smithi063 podman[68683]: 2024-04-25 18:41:36.749065234 +0000 UTC m=+0.193576144 container create 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-25T18:41:36.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.897+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09281a8150 con 0x7f0928103450 2024-04-25T18:41:36.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.898+0000 7f091f400700 1 -- 172.21.15.63:0/287756841 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f091409c8e0 con 0x7f0928103450 2024-04-25T18:41:36.899 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:36.899 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 msgr2=0x7f090c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 0x7f090c0a6180 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0918009fd0 tx=0x7f0918009260 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 msgr2=0x7f092819b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f091400ec10 tx=0x7f091400ef20 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 shutdown_connections 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f090c0a3cc0 0x7f090c0a6180 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0928108a50 0x7f092819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0928102a90 0x7f092819b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 --2- 172.21.15.63:0/287756841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0928103450 0x7f092819b880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:36.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 >> 172.21.15.63:0/287756841 conn(0x7f0928079ba0 msgr2=0x7f092810df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:36.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 shutdown_connections 2024-04-25T18:41:36.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:36.901+0000 7f092dc00700 1 -- 172.21.15.63:0/287756841 wait complete. 2024-04-25T18:41:36.903 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 3 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:37 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/287756841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 podman[68683]: 2024-04-25 18:41:37.24537205 +0000 UTC m=+0.689883000 container init 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 podman[68683]: 2024-04-25 18:41:37.256305918 +0000 UTC m=+0.700816828 container start 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 ceph-mds[68735]: -- 172.21.15.63:0/3927370402 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c03da1a40 con 0x558c03d74c00 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j[68717]: starting mds.j at 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 bash[68683]: 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f 2024-04-25T18:41:37.540 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 18:41:37 smithi063 systemd[1]: Started Ceph mds.j for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:37 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/287756841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:37.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:37 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/287756841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.63:6834/3714473913,v1:172.21.15.63:6835/3714473913] up:boot 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: fsmap 3 up:standby 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:38 smithi063 ceph-mon[28006]: Deploying daemon mds.d on smithi063 2024-04-25T18:41:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.63:6834/3714473913,v1:172.21.15.63:6835/3714473913] up:boot 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: fsmap 3 up:standby 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:38 smithi070 ceph-mon[32820]: Deploying daemon mds.d on smithi063 2024-04-25T18:41:38.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:38.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.63:6834/3714473913,v1:172.21.15.63:6835/3714473913] up:boot 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: fsmap 3 up:standby 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:38.719 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:38 smithi114 ceph-mon[32988]: Deploying daemon mds.d on smithi063 2024-04-25T18:41:39.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:40.638 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:40 smithi063 ceph-mon[28006]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:40.638 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:40 smithi063 systemd[1]: Starting Ceph mds.d for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:40.656 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:40 smithi070 ceph-mon[32820]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:40.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:40 smithi114 ceph-mon[32988]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:41.923 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:41 smithi063 podman[69119]: 2024-04-25 18:41:41.691089122 +0000 UTC m=+0.188481637 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:42.263 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 podman[69119]: 2024-04-25 18:41:42.130696592 +0000 UTC m=+0.628089123 container create 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T18:41:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:42 smithi063 ceph-mon[28006]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 podman[69119]: 2024-04-25 18:41:42.392587153 +0000 UTC m=+0.889979691 container init 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 podman[69119]: 2024-04-25 18:41:42.405406435 +0000 UTC m=+0.902798952 container start 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 ceph-mds[69162]: -- 172.21.15.63:0/1678243164 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffa16e9a40 con 0x55ffa16bd000 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d[69155]: starting mds.d at 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 bash[69119]: 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 2024-04-25T18:41:42.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 18:41:42 smithi063 systemd[1]: Started Ceph mds.d for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:42 smithi070 ceph-mon[32820]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:42.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:42 smithi114 ceph-mon[32988]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 -- 172.21.15.63:0/2890507942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc570101770 msgr2=0x7fc570101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 --2- 172.21.15.63:0/2890507942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc570101770 0x7fc570101b50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc564009990 tx=0x7fc564009ca0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 -- 172.21.15.63:0/2890507942 shutdown_connections 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 --2- 172.21.15.63:0/2890507942 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc57010f320 0x7fc570111710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 --2- 172.21.15.63:0/2890507942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc570101770 0x7fc570101b50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 --2- 172.21.15.63:0/2890507942 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570102120 0x7fc57010ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 -- 172.21.15.63:0/2890507942 >> 172.21.15.63:0/2890507942 conn(0x7fc5700fd3d0 msgr2=0x7fc5700ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 -- 172.21.15.63:0/2890507942 shutdown_connections 2024-04-25T18:41:43.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.241+0000 7fc575e00700 1 -- 172.21.15.63:0/2890507942 wait complete. 2024-04-25T18:41:43.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.242+0000 7fc575e00700 1 Processor -- start 2024-04-25T18:41:43.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.242+0000 7fc575e00700 1 -- start start 2024-04-25T18:41:43.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:43.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 0x7fc57019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:43.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 0x7fc57019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc570114e00 con 0x7fc57010f320 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc570114c80 con 0x7fc570102120 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc570114f80 con 0x7fc570101770 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc574a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc56fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 0x7fc57019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc574a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45396/0 (socket says 172.21.15.63:45396) 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc574a00700 1 -- 172.21.15.63:0/571572523 learned_addr learned my addr 172.21.15.63:0/571572523 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.243+0000 7fc575400700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 0x7fc57019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 -- 172.21.15.63:0/571572523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 msgr2=0x7fc57019b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 -- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 msgr2=0x7fc57019fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 0x7fc57019fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 -- 172.21.15.63:0/571572523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5701a0360 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc575400700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 0x7fc57019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc574a00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56fe00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 0x7fc57019b890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00d750 tx=0x7fc55c00da60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c016070 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc55c005020 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c00fa10 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5701a0650 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.244+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5701a7ed0 con 0x7fc570102120 2024-04-25T18:41:43.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.245+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc570069730 con 0x7fc570102120 2024-04-25T18:41:43.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.248+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc55c004a30 con 0x7fc570102120 2024-04-25T18:41:43.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.249+0000 7fc56d600700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 0x7fc5580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:43.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.249+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7fc55c0cfa60 con 0x7fc570102120 2024-04-25T18:41:43.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.249+0000 7fc574a00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 0x7fc5580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:43.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.250+0000 7fc574a00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 0x7fc5580a6100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc564006010 tx=0x7fc56400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.254+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55c09c840 con 0x7fc570102120 2024-04-25T18:41:43.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.543+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc57019c6b0 con 0x7fc570102120 2024-04-25T18:41:43.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.544+0000 7fc56d600700 1 -- 172.21.15.63:0/571572523 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fc55c09c140 con 0x7fc570102120 2024-04-25T18:41:43.545 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:43.545 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-25T18:41:43.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 msgr2=0x7fc5580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:43.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 0x7fc5580a6100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc564006010 tx=0x7fc56400b280 comp rx=0 tx=0).stop 2024-04-25T18:41:43.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 msgr2=0x7fc57019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:43.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 0x7fc57019b890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc55c00d750 tx=0x7fc55c00da60 comp rx=0 tx=0).stop 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 shutdown_connections 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5580a3c40 0x7fc5580a6100 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc570101770 0x7fc57019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc57010f320 0x7fc57019fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 --2- 172.21.15.63:0/571572523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc570102120 0x7fc57019b890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 >> 172.21.15.63:0/571572523 conn(0x7fc5700fd3d0 msgr2=0x7fc5700ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 shutdown_connections 2024-04-25T18:41:43.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:43.547+0000 7fc575e00700 1 -- 172.21.15.63:0/571572523 wait complete. 2024-04-25T18:41:43.550 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 5 2024-04-25T18:41:43.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: Deploying daemon mds.e on smithi070 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.63:6836/166123931,v1:172.21.15.63:6837/166123931] up:boot 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: fsmap 4 up:standby 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:41:43.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:43 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/571572523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: Deploying daemon mds.e on smithi070 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.63:6836/166123931,v1:172.21.15.63:6837/166123931] up:boot 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: fsmap 4 up:standby 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:41:43.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:43 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/571572523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: Deploying daemon mds.e on smithi070 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.63:6836/166123931,v1:172.21.15.63:6837/166123931] up:boot 2024-04-25T18:41:44.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: fsmap 4 up:standby 2024-04-25T18:41:44.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:41:44.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:43 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/571572523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:44.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:44 smithi070 ceph-mon[32820]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:44.954 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:44 smithi070 systemd[1]: Starting Ceph mds.e for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:44 smithi114 ceph-mon[32988]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:45.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:44 smithi063 ceph-mon[28006]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:45.277 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 podman[62884]: 2024-04-25 18:41:44.909574565 +0000 UTC m=+0.018059745 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:45.277 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 podman[62884]: 2024-04-25 18:41:45.04927773 +0000 UTC m=+0.157762887 container create 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T18:41:45.277 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 podman[62884]: 2024-04-25 18:41:45.276453519 +0000 UTC m=+0.384938682 container init 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T18:41:45.703 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 podman[62884]: 2024-04-25 18:41:45.279063478 +0000 UTC m=+0.387548636 container start 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T18:41:45.703 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 ceph-mds[62903]: -- 172.21.15.70:0/1179599132 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56327da15a40 con 0x56327d9e9000 2024-04-25T18:41:45.704 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: starting mds.e at 2024-04-25T18:41:45.704 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 bash[62884]: 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 2024-04-25T18:41:45.704 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 18:41:45 smithi070 systemd[1]: Started Ceph mds.e for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:46.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: Deploying daemon mds.f on smithi114 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6834/3182056159,v1:172.21.15.70:6835/3182056159] up:boot 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: fsmap 5 up:standby 2024-04-25T18:41:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:46 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:46.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: Deploying daemon mds.f on smithi114 2024-04-25T18:41:46.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6834/3182056159,v1:172.21.15.70:6835/3182056159] up:boot 2024-04-25T18:41:46.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: fsmap 5 up:standby 2024-04-25T18:41:46.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:46 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: Deploying daemon mds.f on smithi114 2024-04-25T18:41:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6834/3182056159,v1:172.21.15.70:6835/3182056159] up:boot 2024-04-25T18:41:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: fsmap 5 up:standby 2024-04-25T18:41:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:46 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:41:46.994 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:47.817 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:47 smithi114 systemd[1]: Starting Ceph mds.f for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:48.121 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:47 smithi114 podman[63087]: 2024-04-25 18:41:47.763539249 +0000 UTC m=+0.017631271 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:48.121 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:47 smithi114 podman[63087]: 2024-04-25 18:41:47.89379914 +0000 UTC m=+0.147891148 container create c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 podman[63087]: 2024-04-25 18:41:48.120744385 +0000 UTC m=+0.374836393 container init c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 podman[63087]: 2024-04-25 18:41:48.123452631 +0000 UTC m=+0.377544644 container start c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5) 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/156154363 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55796fffda40 con 0x55796ffd0c00 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 bash[63087]: c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 2024-04-25T18:41:48.438 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:41:48 smithi114 systemd[1]: Started Ceph mds.f for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:48 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:48.727 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:48 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:48 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:49.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 -- 172.21.15.63:0/356424300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2304108a30 msgr2=0x7f2304108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 --2- 172.21.15.63:0/356424300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2304108a30 0x7f2304108e10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098f0 tx=0x7f22f4009c00 comp rx=0 tx=0).stop 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 -- 172.21.15.63:0/356424300 shutdown_connections 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 --2- 172.21.15.63:0/356424300 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2304108a30 0x7f2304108e10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 --2- 172.21.15.63:0/356424300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406b0f0 0x7f230410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 --2- 172.21.15.63:0/356424300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f230406a750 0x7f230406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 -- 172.21.15.63:0/356424300 >> 172.21.15.63:0/356424300 conn(0x7f23040788f0 msgr2=0x7f2304078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 -- 172.21.15.63:0/356424300 shutdown_connections 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.237+0000 7f2309a00700 1 -- 172.21.15.63:0/356424300 wait complete. 2024-04-25T18:41:49.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.239+0000 7f2309a00700 1 Processor -- start 2024-04-25T18:41:49.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.239+0000 7f2309a00700 1 -- start start 2024-04-25T18:41:49.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 0x7f2304109e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2304108a30 0x7f230410a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2304112bd0 con 0x7f2304108a30 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2304112a50 con 0x7f230406a750 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2309a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2304112d50 con 0x7f230406b0f0 2024-04-25T18:41:49.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 0x7f2304109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2303400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2304108a30 0x7f230410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 0x7f2304109e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41068/0 (socket says 172.21.15.63:41068) 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47840/0 (socket says 172.21.15.63:47840) 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302000700 1 -- 172.21.15.63:0/2612993665 learned_addr learned my addr 172.21.15.63:0/2612993665 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 -- 172.21.15.63:0/2612993665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 msgr2=0x7f2304109e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 0x7f2304109e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 -- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2304108a30 msgr2=0x7f230410a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2304108a30 0x7f230410a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.240+0000 7f2302a00700 1 -- 172.21.15.63:0/2612993665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23041abab0 con 0x7f230406a750 2024-04-25T18:41:49.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f2302a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098f0 tx=0x7f22f400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4025070 con 0x7f230406a750 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23041abd40 con 0x7f230406a750 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22f400b560 con 0x7f230406a750 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4003900 con 0x7f230406a750 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.241+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23041ac190 con 0x7f230406a750 2024-04-25T18:41:49.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.242+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2304104170 con 0x7f230406a750 2024-04-25T18:41:49.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.244+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f22f4028070 con 0x7f230406a750 2024-04-25T18:41:49.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.246+0000 7f22fb400700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 0x7f22e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:49.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.246+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f22f40d08e0 con 0x7f230406a750 2024-04-25T18:41:49.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.246+0000 7f2302000700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 0x7f22e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:49.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.247+0000 7f2302000700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 0x7f22e80a6100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f230410b0d0 tx=0x7f22f0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:49.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.248+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22f4013030 con 0x7f230406a750 2024-04-25T18:41:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.534+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f230402e140 con 0x7f230406a750 2024-04-25T18:41:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.535+0000 7f22fb400700 1 -- 172.21.15.63:0/2612993665 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f22f409d750 con 0x7f230406a750 2024-04-25T18:41:49.537 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:49.537 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T18:41:49.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.538+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 msgr2=0x7f22e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:49.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 0x7f22e80a6100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f230410b0d0 tx=0x7f22f0009310 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 msgr2=0x7f2304109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f22f40098f0 tx=0x7f22f400fa00 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 shutdown_connections 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f22e80a3c40 0x7f22e80a6100 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2304108a30 0x7f230410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f230406a750 0x7f2304109910 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 --2- 172.21.15.63:0/2612993665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f230406b0f0 0x7f2304109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:49.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 >> 172.21.15.63:0/2612993665 conn(0x7f23040788f0 msgr2=0x7f230410df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:49.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 shutdown_connections 2024-04-25T18:41:49.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:49.539+0000 7f2309a00700 1 -- 172.21.15.63:0/2612993665 wait complete. 2024-04-25T18:41:49.541 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 7 2024-04-25T18:41:49.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:49 smithi114 ceph-mon[32988]: Deploying daemon mds.c on smithi114 2024-04-25T18:41:49.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:49 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/3172988525,v1:172.21.15.114:6835/3172988525] up:boot 2024-04-25T18:41:49.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:49 smithi114 ceph-mon[32988]: fsmap 6 up:standby 2024-04-25T18:41:49.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:49 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:41:49.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:49 smithi070 ceph-mon[32820]: Deploying daemon mds.c on smithi114 2024-04-25T18:41:49.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:49 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/3172988525,v1:172.21.15.114:6835/3172988525] up:boot 2024-04-25T18:41:49.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:49 smithi070 ceph-mon[32820]: fsmap 6 up:standby 2024-04-25T18:41:49.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:49 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:41:49.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:49 smithi063 ceph-mon[28006]: Deploying daemon mds.c on smithi114 2024-04-25T18:41:49.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:49 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/3172988525,v1:172.21.15.114:6835/3172988525] up:boot 2024-04-25T18:41:49.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:49 smithi063 ceph-mon[28006]: fsmap 6 up:standby 2024-04-25T18:41:49.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:49 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:41:50.459 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 systemd[1]: Starting Ceph mds.c for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:50 smithi063 ceph-mon[28006]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:50.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:50 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2612993665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:50.726 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:50 smithi114 ceph-mon[32988]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:50.726 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:50 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2612993665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:50.726 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 podman[63395]: 2024-04-25 18:41:50.479202491 +0000 UTC m=+0.017966008 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:50.726 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 podman[63395]: 2024-04-25 18:41:50.59686841 +0000 UTC m=+0.135631926 container create 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T18:41:50.797 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:50 smithi070 ceph-mon[32820]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:50.797 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:50 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2612993665' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 podman[63395]: 2024-04-25 18:41:50.824418359 +0000 UTC m=+0.363181877 container init 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 podman[63395]: 2024-04-25 18:41:50.827212215 +0000 UTC m=+0.365975732 container start 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 ceph-mds[63414]: -- 172.21.15.114:0/4022959559 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd31b99a40 con 0x55fd31b6cc00 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: starting mds.c at 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 bash[63395]: 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f 2024-04-25T18:41:51.317 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 18:41:50 smithi114 systemd[1]: Started Ceph mds.c for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: Deploying daemon mds.h on smithi070 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:boot 2024-04-25T18:41:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: fsmap 7 up:standby 2024-04-25T18:41:52.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:51 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: Deploying daemon mds.h on smithi070 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:boot 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: fsmap 7 up:standby 2024-04-25T18:41:52.321 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:51 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: Deploying daemon mds.h on smithi070 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:boot 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: fsmap 7 up:standby 2024-04-25T18:41:52.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:51 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:41:52.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:52.876 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:41:53.203 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 systemd[1]: Starting Ceph mds.h for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:53.560 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 podman[63201]: 2024-04-25 18:41:53.196141414 +0000 UTC m=+0.017480422 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:53.560 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 podman[63201]: 2024-04-25 18:41:53.314646547 +0000 UTC m=+0.135985586 container create 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:41:53.560 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 podman[63201]: 2024-04-25 18:41:53.533814652 +0000 UTC m=+0.355153647 container init 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T18:41:53.560 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 podman[63201]: 2024-04-25 18:41:53.53639764 +0000 UTC m=+0.357736634 container start 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T18:41:53.953 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 ceph-mds[63221]: -- 172.21.15.70:0/138557680 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ac2a67a40 con 0x564ac2a3ac00 2024-04-25T18:41:53.953 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: starting mds.h at 2024-04-25T18:41:53.954 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 bash[63201]: 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 2024-04-25T18:41:53.954 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 18:41:53 smithi070 systemd[1]: Started Ceph mds.h for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:54.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:54.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: Deploying daemon mds.i on smithi114 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6836/4043589496,v1:172.21.15.70:6837/4043589496] up:boot 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: fsmap 8 up:standby 2024-04-25T18:41:54.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:41:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: Deploying daemon mds.i on smithi114 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6836/4043589496,v1:172.21.15.70:6837/4043589496] up:boot 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: fsmap 8 up:standby 2024-04-25T18:41:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:41:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: Deploying daemon mds.i on smithi114 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6836/4043589496,v1:172.21.15.70:6837/4043589496] up:boot 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: fsmap 8 up:standby 2024-04-25T18:41:55.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:41:55.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 -- 172.21.15.63:0/3706765054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006b0f0 msgr2=0x7f34d010f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 --2- 172.21.15.63:0/3706765054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006b0f0 0x7f34d010f900 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009f90 tx=0x7f34c800b330 comp rx=0 tx=0).stop 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 -- 172.21.15.63:0/3706765054 shutdown_connections 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 --2- 172.21.15.63:0/3706765054 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 0x7f34d0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 --2- 172.21.15.63:0/3706765054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006b0f0 0x7f34d010f900 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 --2- 172.21.15.63:0/3706765054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006a750 0x7f34d006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 -- 172.21.15.63:0/3706765054 >> 172.21.15.63:0/3706765054 conn(0x7f34d00788f0 msgr2=0x7f34d0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 -- 172.21.15.63:0/3706765054 shutdown_connections 2024-04-25T18:41:55.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.377+0000 7f34d5400700 1 -- 172.21.15.63:0/3706765054 wait complete. 2024-04-25T18:41:55.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.378+0000 7f34d5400700 1 Processor -- start 2024-04-25T18:41:55.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.378+0000 7f34d5400700 1 -- start start 2024-04-25T18:41:55.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 0x7f34d019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 0x7f34d019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0112c20 con 0x7f34d006a750 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0112aa0 con 0x7f34d006b0f0 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34d5400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f34d0112da0 con 0x7f34d0108a30 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34ce800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34ce800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33732/0 (socket says 172.21.15.63:33732) 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34ce800700 1 -- 172.21.15.63:0/3473127883 learned_addr learned my addr 172.21.15.63:0/3473127883 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.379+0000 7f34c5e00700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 0x7f34d019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:55.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34cf200700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 0x7f34d019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 -- 172.21.15.63:0/3473127883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 msgr2=0x7f34d019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 0x7f34d019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 -- 172.21.15.63:0/3473127883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 msgr2=0x7f34d019b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 0x7f34d019b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 -- 172.21.15.63:0/3473127883 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d01a0310 con 0x7f34d006a750 2024-04-25T18:41:55.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34c5e00700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 0x7f34d019b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.380+0000 7f34ce800700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34c000e9c0 tx=0x7f34c000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.381+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0004a30 con 0x7f34d006a750 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.381+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34c000cc70 con 0x7f34d006a750 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.381+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d01a0600 con 0x7f34d006a750 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.381+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c0005490 con 0x7f34d006a750 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.381+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d01a7e80 con 0x7f34d006a750 2024-04-25T18:41:55.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.382+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d0069730 con 0x7f34d006a750 2024-04-25T18:41:55.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.384+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f34c0004bd0 con 0x7f34d006a750 2024-04-25T18:41:55.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.386+0000 7f34c7e00700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 0x7f34b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:41:55.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.386+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f34c00cffa0 con 0x7f34d006a750 2024-04-25T18:41:55.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.387+0000 7f34c5e00700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 0x7f34b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:41:55.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.387+0000 7f34c5e00700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 0x7f34b40a6180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f34d0078ef0 tx=0x7f34b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:41:55.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.391+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34c009cd80 con 0x7f34d006a750 2024-04-25T18:41:55.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.672+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34d002df80 con 0x7f34d006a750 2024-04-25T18:41:55.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.673+0000 7f34c7e00700 1 -- 172.21.15.63:0/3473127883 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f34c009c680 con 0x7f34d006a750 2024-04-25T18:41:55.674 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:41:55.674 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T18:41:55.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 msgr2=0x7f34b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:55.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 0x7f34b40a6180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f34d0078ef0 tx=0x7f34b8009260 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 msgr2=0x7f34d019b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f34c000e9c0 tx=0x7f34c000ecd0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 shutdown_connections 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f34b40a3cc0 0x7f34b40a6180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f34d0108a30 0x7f34d019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34d006a750 0x7f34d019b350 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 --2- 172.21.15.63:0/3473127883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d006b0f0 0x7f34d019b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.676+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 >> 172.21.15.63:0/3473127883 conn(0x7f34d00788f0 msgr2=0x7f34d010df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.677+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 shutdown_connections 2024-04-25T18:41:55.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:41:55.677+0000 7f34d5400700 1 -- 172.21.15.63:0/3473127883 wait complete. 2024-04-25T18:41:55.679 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 9 2024-04-25T18:41:55.817 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:55 smithi114 systemd[1]: Starting Ceph mds.i for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:41:56.244 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:55 smithi114 podman[63702]: 2024-04-25 18:41:55.924287037 +0000 UTC m=+0.109563212 container create 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:55 smithi114 podman[63702]: 2024-04-25 18:41:55.832557141 +0000 UTC m=+0.017833339 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 podman[63702]: 2024-04-25 18:41:56.118225016 +0000 UTC m=+0.303501190 container init 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 podman[63702]: 2024-04-25 18:41:56.120788014 +0000 UTC m=+0.306064194 container start 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mds[63721]: -- 172.21.15.114:0/1971901356 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645185cba40 con 0x56451859e800 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: starting mds.i at 2024-04-25T18:41:56.245 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 bash[63702]: 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 2024-04-25T18:41:56.567 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 18:41:56 smithi114 systemd[1]: Started Ceph mds.i for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3473127883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3473127883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3473127883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:41:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:41:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:57 smithi114 ceph-mon[32988]: Deploying daemon mds.g on smithi063 2024-04-25T18:41:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:57 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:boot 2024-04-25T18:41:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:57 smithi114 ceph-mon[32988]: fsmap 9 up:standby 2024-04-25T18:41:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:41:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:57 smithi070 ceph-mon[32820]: Deploying daemon mds.g on smithi063 2024-04-25T18:41:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:57 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:boot 2024-04-25T18:41:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:57 smithi070 ceph-mon[32820]: fsmap 9 up:standby 2024-04-25T18:41:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:57 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:41:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:57 smithi063 ceph-mon[28006]: Deploying daemon mds.g on smithi063 2024-04-25T18:41:58.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:57 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:boot 2024-04-25T18:41:58.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:57 smithi063 ceph-mon[28006]: fsmap 9 up:standby 2024-04-25T18:41:58.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:41:58.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:41:58.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:41:58 smithi114 ceph-mon[32988]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:41:58 smithi063 ceph-mon[28006]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:41:58 smithi070 ceph-mon[32820]: pgmap v317: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:41:59.016 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:00.836 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:00 smithi063 ceph-mon[28006]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:00 smithi070 ceph-mon[32820]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:01.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:00 smithi114 ceph-mon[32988]: pgmap v318: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:01.377 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 18:42:01 smithi063 podman[69891]: 2024-04-25 18:42:01.368856527 +0000 UTC m=+0.664832609 container start 750410dee27fe8074654c22e23d8564f96c70f73cb8e9b2f946bde7709a0fd1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:42:01.703 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 18:42:01 smithi063 ceph-mds[69933]: -- 172.21.15.63:0/1511703221 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56341fc3da40 con 0x56341fc10800 2024-04-25T18:42:01.703 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 18:42:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g[69924]: starting mds.g at 2024-04-25T18:42:01.703 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 18:42:01 smithi063 bash[69891]: 750410dee27fe8074654c22e23d8564f96c70f73cb8e9b2f946bde7709a0fd1b 2024-04-25T18:42:01.703 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 18:42:01 smithi063 systemd[1]: Started Ceph mds.g for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:42:01.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 -- 172.21.15.63:0/1862231526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4075590 msgr2=0x7f99a410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:01.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 --2- 172.21.15.63:0/1862231526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4075590 0x7f99a410ef00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9990009990 tx=0x7f9990009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 -- 172.21.15.63:0/1862231526 shutdown_connections 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 --2- 172.21.15.63:0/1862231526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a4074c70 0x7f99a4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 --2- 172.21.15.63:0/1862231526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a410f440 0x7f99a4111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 --2- 172.21.15.63:0/1862231526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4075590 0x7f99a410ef00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 -- 172.21.15.63:0/1862231526 >> 172.21.15.63:0/1862231526 conn(0x7f99a4079ba0 msgr2=0x7f99a4079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 -- 172.21.15.63:0/1862231526 shutdown_connections 2024-04-25T18:42:01.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.743+0000 7f99abe00700 1 -- 172.21.15.63:0/1862231526 wait complete. 2024-04-25T18:42:01.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.744+0000 7f99abe00700 1 Processor -- start 2024-04-25T18:42:01.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.744+0000 7f99abe00700 1 -- start start 2024-04-25T18:42:01.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4074c70 0x7f99a419b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:01.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 0x7f99a419fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f99a4114e50 con 0x7f99a4074c70 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99a4114cd0 con 0x7f99a4075590 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99abe00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f99a4114fd0 con 0x7f99a410f440 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99a8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99a9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4074c70 0x7f99a419b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99a8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:37688/0 (socket says 172.21.15.63:37688) 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99a8a00700 1 -- 172.21.15.63:0/4114022787 learned_addr learned my addr 172.21.15.63:0/4114022787 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.745+0000 7f99a9e00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 0x7f99a419fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:01.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 -- 172.21.15.63:0/4114022787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 msgr2=0x7f99a419fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 0x7f99a419fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 -- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4074c70 msgr2=0x7f99a419b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4074c70 0x7f99a419b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 -- 172.21.15.63:0/4114022787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a41a0390 con 0x7f99a4075590 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a9e00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 0x7f99a419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:01.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a8a00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9990000c00 tx=0x7f99900057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990027070 con 0x7f99a4075590 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9990022470 con 0x7f99a4075590 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.746+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a41a0620 con 0x7f99a4075590 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.747+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990017650 con 0x7f99a4075590 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.747+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a41a7f00 con 0x7f99a4075590 2024-04-25T18:42:01.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.748+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a4076360 con 0x7f99a4075590 2024-04-25T18:42:01.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.749+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f999002a070 con 0x7f99a4075590 2024-04-25T18:42:01.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.751+0000 7f99a2000700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 0x7f998c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:01.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.751+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f99900d0c50 con 0x7f99a4075590 2024-04-25T18:42:01.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.751+0000 7f99a9400700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 0x7f998c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:01.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.752+0000 7f99a9400700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 0x7f998c0a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9998009c60 tx=0x7f9998009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:01.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:01.755+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9990013070 con 0x7f99a4075590 2024-04-25T18:42:02.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.043+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99a402df80 con 0x7f99a4075590 2024-04-25T18:42:02.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.044+0000 7f99a2000700 1 -- 172.21.15.63:0/4114022787 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7f999009dac0 con 0x7f99a4075590 2024-04-25T18:42:02.045 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:02.045 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.048+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 msgr2=0x7f998c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.048+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 0x7f998c0a6100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9998009c60 tx=0x7f9998009260 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.048+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 msgr2=0x7f99a419b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.048+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9990000c00 tx=0x7f99900057c0 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 shutdown_connections 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f998c0a3c40 0x7f998c0a6100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f99a410f440 0x7f99a419fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99a4074c70 0x7f99a419b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 --2- 172.21.15.63:0/4114022787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99a4075590 0x7f99a419b8c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:02.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 >> 172.21.15.63:0/4114022787 conn(0x7f99a4079ba0 msgr2=0x7f99a40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:02.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 shutdown_connections 2024-04-25T18:42:02.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:02.049+0000 7f99abe00700 1 -- 172.21.15.63:0/4114022787 wait complete. 2024-04-25T18:42:02.052 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 11 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: Deploying daemon mds.a on smithi063 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.63:6838/3937073235,v1:172.21.15.63:6839/3937073235] up:boot 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: fsmap 10 up:standby 2024-04-25T18:42:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:42:02.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:02 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4114022787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:02.980 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: Deploying daemon mds.a on smithi063 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.63:6838/3937073235,v1:172.21.15.63:6839/3937073235] up:boot 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: fsmap 10 up:standby 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:42:02.981 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:02 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4114022787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: pgmap v319: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: Deploying daemon mds.a on smithi063 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.63:6838/3937073235,v1:172.21.15.63:6839/3937073235] up:boot 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: fsmap 10 up:standby 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:42:03.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:02 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4114022787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:04 smithi063 ceph-mon[28006]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:04 smithi070 ceph-mon[32820]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:04.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:42:05.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:04 smithi114 ceph-mon[32988]: pgmap v320: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:05.627 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:05 smithi063 systemd[1]: Starting Ceph mds.a for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:42:06.022 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:05 smithi063 podman[70296]: 2024-04-25 18:42:05.599299814 +0000 UTC m=+0.037096300 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:42:06.023 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:05 smithi063 podman[70296]: 2024-04-25 18:42:05.716910482 +0000 UTC m=+0.154707055 container create 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-25T18:42:06.023 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 podman[70296]: 2024-04-25 18:42:06.020985433 +0000 UTC m=+0.458781963 container init 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T18:42:06.181 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:06.337 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 podman[70296]: 2024-04-25 18:42:06.031833188 +0000 UTC m=+0.469629691 container start 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-25T18:42:06.337 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mds[70336]: -- 172.21.15.63:0/2355250566 <== mon.0 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628747dfa40 con 0x5628747b2800 2024-04-25T18:42:06.337 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a[70332]: starting mds.a at 2024-04-25T18:42:06.337 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 bash[70296]: 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 2024-04-25T18:42:06.337 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 18:42:06 smithi063 systemd[1]: Started Ceph mds.a for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:06 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:06.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:06.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:06 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: pgmap v321: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T18:42:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:06 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:07.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:07 smithi063 ceph-mon[28006]: Deploying daemon mds.b on smithi070 2024-04-25T18:42:07.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:07 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.63:6840/512888768,v1:172.21.15.63:6841/512888768] up:boot 2024-04-25T18:42:07.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:07 smithi063 ceph-mon[28006]: fsmap 11 up:standby 2024-04-25T18:42:07.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:07 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:42:07.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:07 smithi070 ceph-mon[32820]: Deploying daemon mds.b on smithi070 2024-04-25T18:42:07.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:07 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.63:6840/512888768,v1:172.21.15.63:6841/512888768] up:boot 2024-04-25T18:42:07.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:07 smithi070 ceph-mon[32820]: fsmap 11 up:standby 2024-04-25T18:42:07.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:07 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:42:07.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:07 smithi114 ceph-mon[32988]: Deploying daemon mds.b on smithi070 2024-04-25T18:42:07.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:07 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.63:6840/512888768,v1:172.21.15.63:6841/512888768] up:boot 2024-04-25T18:42:07.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:07 smithi114 ceph-mon[32988]: fsmap 11 up:standby 2024-04-25T18:42:07.991 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:07 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 -- 172.21.15.63:0/1523619423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44074de0 msgr2=0x7faf440751c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 --2- 172.21.15.63:0/1523619423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44074de0 0x7faf440751c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf34009f90 tx=0x7faf3400b330 comp rx=0 tx=0).stop 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 -- 172.21.15.63:0/1523619423 shutdown_connections 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 --2- 172.21.15.63:0/1523619423 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44074de0 0x7faf440751c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 --2- 172.21.15.63:0/1523619423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf44115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 --2- 172.21.15.63:0/1523619423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44075700 0x7faf44113440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 -- 172.21.15.63:0/1523619423 >> 172.21.15.63:0/1523619423 conn(0x7faf440fe4c0 msgr2=0x7faf441008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 -- 172.21.15.63:0/1523619423 shutdown_connections 2024-04-25T18:42:08.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.485+0000 7faf43200700 1 -- 172.21.15.63:0/1523619423 wait complete. 2024-04-25T18:42:08.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.487+0000 7faf43200700 1 Processor -- start 2024-04-25T18:42:08.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 -- start start 2024-04-25T18:42:08.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:08.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 0x7faf4419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf441a3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faf44119150 con 0x7faf44074de0 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faf44118fd0 con 0x7faf44113980 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.488+0000 7faf43200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faf441192d0 con 0x7faf44075700 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 0x7faf4419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf41e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46504/0 (socket says 172.21.15.63:46504) 2024-04-25T18:42:08.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf41e00700 1 -- 172.21.15.63:0/945002561 learned_addr learned my addr 172.21.15.63:0/945002561 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf42800700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf441a3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 -- 172.21.15.63:0/945002561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 msgr2=0x7faf441a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf441a3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 -- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 msgr2=0x7faf4419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf39400700 1 -- 172.21.15.63:0/945002561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf441a4650 con 0x7faf44075700 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf42800700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf441a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.489+0000 7faf41e00700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf39400700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 0x7faf4419fb80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e9f0 tx=0x7faf2c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c00c9c0 con 0x7faf44075700 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf441a4940 con 0x7faf44075700 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf2c00cb60 con 0x7faf44075700 2024-04-25T18:42:08.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf2c018680 con 0x7faf44075700 2024-04-25T18:42:08.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.490+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf441ac1c0 con 0x7faf44075700 2024-04-25T18:42:08.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.491+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf44069730 con 0x7faf44075700 2024-04-25T18:42:08.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.493+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faf2c018a30 con 0x7faf44075700 2024-04-25T18:42:08.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.495+0000 7faf3b400700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 0x7faf280a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:08.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.495+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7faf2c0d0420 con 0x7faf44075700 2024-04-25T18:42:08.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.495+0000 7faf41e00700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 0x7faf280a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:08.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.496+0000 7faf41e00700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 0x7faf280a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faf34009790 tx=0x7faf340096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:08.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.497+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf2c09d200 con 0x7faf44075700 2024-04-25T18:42:08.615 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 systemd[1]: Starting Ceph mds.b for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T18:42:08.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.780+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf441a09a0 con 0x7faf44075700 2024-04-25T18:42:08.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:08.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.781+0000 7faf3b400700 1 -- 172.21.15.63:0/945002561 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7faf2c09cb00 con 0x7faf44075700 2024-04-25T18:42:08.783 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T18:42:08.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 msgr2=0x7faf280a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:08.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 0x7faf280a6180 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faf34009790 tx=0x7faf340096c0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 msgr2=0x7faf4419fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:08.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 0x7faf4419fb80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf2c00e9f0 tx=0x7faf2c00ed00 comp rx=0 tx=0).stop 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 shutdown_connections 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faf280a3cc0 0x7faf280a6180 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faf44113980 0x7faf441a3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf44074de0 0x7faf4419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 --2- 172.21.15.63:0/945002561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faf44075700 0x7faf4419fb80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 >> 172.21.15.63:0/945002561 conn(0x7faf440fe4c0 msgr2=0x7faf440ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 shutdown_connections 2024-04-25T18:42:08.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:08.784+0000 7faf43200700 1 -- 172.21.15.63:0/945002561 wait complete. 2024-04-25T18:42:08.787 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 12 2024-04-25T18:42:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:08 smithi063 ceph-mon[28006]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:08.951 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:08 smithi070 ceph-mon[32820]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:08.951 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 podman[63511]: 2024-04-25 18:42:08.615461573 +0000 UTC m=+0.017857349 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T18:42:08.951 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 podman[63511]: 2024-04-25 18:42:08.715152922 +0000 UTC m=+0.117548679 container create b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T18:42:08.951 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 podman[63511]: 2024-04-25 18:42:08.950920261 +0000 UTC m=+0.353316026 container init b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-25T18:42:09.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:08 smithi114 ceph-mon[32988]: pgmap v322: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:09.203 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 podman[63511]: 2024-04-25 18:42:08.953612454 +0000 UTC m=+0.356008211 container start b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T18:42:09.203 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 ceph-mds[63530]: -- 172.21.15.70:0/3594699777 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612c2bb3a40 con 0x5612c2b86c00 2024-04-25T18:42:09.203 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: starting mds.b at 2024-04-25T18:42:09.204 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:09 smithi070 bash[63511]: b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 2024-04-25T18:42:09.204 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:09 smithi070 systemd[1]: Started Ceph mds.b for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T18:42:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/945002561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:09 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/945002561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:09.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:09 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/945002561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:09 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:10.802 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:10 smithi070 ceph-mon[32820]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:10.802 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:10 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:boot 2024-04-25T18:42:10.802 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:10 smithi070 ceph-mon[32820]: fsmap 12 up:standby 2024-04-25T18:42:10.802 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:10 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:42:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:10 smithi063 ceph-mon[28006]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:10 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:boot 2024-04-25T18:42:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:10 smithi063 ceph-mon[28006]: fsmap 12 up:standby 2024-04-25T18:42:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:10 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:42:11.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:10 smithi114 ceph-mon[32988]: pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:10 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:boot 2024-04-25T18:42:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:10 smithi114 ceph-mon[32988]: fsmap 12 up:standby 2024-04-25T18:42:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:10 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:42:11.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:42:12.225 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:12.376 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:12 smithi063 ceph-mon[28006]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:12 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:12 smithi070 ceph-mon[32820]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:12 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:12 smithi114 ceph-mon[32988]: pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:12 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:13.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:13 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:13.657 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:13 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.674 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:13.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:13.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:13.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:42:13.675 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:13 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:14.462 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:14 smithi063 ceph-mon[28006]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:14 smithi070 ceph-mon[32820]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:14 smithi114 ceph-mon[32988]: pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 -- 172.21.15.63:0/1737762958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4105760 msgr2=0x7f48d41097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 --2- 172.21.15.63:0/1737762958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4105760 0x7f48d41097b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48c00099c0 tx=0x7f48c0009cd0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 -- 172.21.15.63:0/1737762958 shutdown_connections 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 --2- 172.21.15.63:0/1737762958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4109e90 0x7f48d4111a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 --2- 172.21.15.63:0/1737762958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4104db0 0x7f48d4105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 --2- 172.21.15.63:0/1737762958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4105760 0x7f48d41097b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 -- 172.21.15.63:0/1737762958 >> 172.21.15.63:0/1737762958 conn(0x7f48d4100620 msgr2=0x7f48d4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 -- 172.21.15.63:0/1737762958 shutdown_connections 2024-04-25T18:42:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.817+0000 7f48dac00700 1 -- 172.21.15.63:0/1737762958 wait complete. 2024-04-25T18:42:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.818+0000 7f48dac00700 1 Processor -- start 2024-04-25T18:42:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 -- start start 2024-04-25T18:42:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 0x7f48d419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4109e90 0x7f48d419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4114d70 con 0x7f48d4105760 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4114bf0 con 0x7f48d4104db0 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48dac00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48d4114ef0 con 0x7f48d4109e90 2024-04-25T18:42:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48d9800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.819+0000 7f48d9800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35260/0 (socket says 172.21.15.63:35260) 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48da200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4109e90 0x7f48d419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 -- 172.21.15.63:0/218072380 learned_addr learned my addr 172.21.15.63:0/218072380 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d8e00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 0x7f48d419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 -- 172.21.15.63:0/218072380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4109e90 msgr2=0x7f48d419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:14.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4109e90 0x7f48d419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 -- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 msgr2=0x7f48d419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 0x7f48d419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 -- 172.21.15.63:0/218072380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d41a02e0 con 0x7f48d4104db0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d8e00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 0x7f48d419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d9800700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48c800eb30 tx=0x7f48c800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c800cc90 con 0x7f48d4104db0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.820+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48c80044d0 con 0x7f48d4104db0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.821+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48d41a05d0 con 0x7f48d4104db0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.821+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48c8010430 con 0x7f48d4104db0 2024-04-25T18:42:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.821+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d41a7e50 con 0x7f48d4104db0 2024-04-25T18:42:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.822+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d4069730 con 0x7f48d4104db0 2024-04-25T18:42:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.824+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f48c8003ea0 con 0x7f48d4104db0 2024-04-25T18:42:14.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.825+0000 7f48d2000700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 0x7f48bc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:14.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.825+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f48c80d01c0 con 0x7f48d4104db0 2024-04-25T18:42:14.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.825+0000 7f48d8e00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 0x7f48bc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:14.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.826+0000 7f48d8e00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 0x7f48bc0a6180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000c00 tx=0x7f48c0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:14.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:14.829+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48c809cfa0 con 0x7f48d4104db0 2024-04-25T18:42:15.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.121+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48d419c630 con 0x7f48d4104db0 2024-04-25T18:42:15.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.122+0000 7f48d2000700 1 -- 172.21.15.63:0/218072380 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f48c809c8a0 con 0x7f48d4104db0 2024-04-25T18:42:15.124 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:15.124 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24496,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24508,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24520,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24533,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T18:42:15.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 msgr2=0x7f48bc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 0x7f48bc0a6180 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f48c0000c00 tx=0x7f48c0005fd0 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 msgr2=0x7f48d419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f48c800eb30 tx=0x7f48c800ee40 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 shutdown_connections 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f48bc0a3cc0 0x7f48bc0a6180 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f48d4109e90 0x7f48d419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48d4104db0 0x7f48d419b320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 --2- 172.21.15.63:0/218072380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48d4105760 0x7f48d419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 >> 172.21.15.63:0/218072380 conn(0x7f48d4100620 msgr2=0x7f48d41029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 shutdown_connections 2024-04-25T18:42:15.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:15.125+0000 7f48dac00700 1 -- 172.21.15.63:0/218072380 wait complete. 2024-04-25T18:42:15.129 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 13 2024-04-25T18:42:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:15 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:15 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/218072380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:15.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:15 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:15 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/218072380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:15 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:15 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/218072380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:42:16.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-04-25T18:42:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:16 smithi063 ceph-mon[28006]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:16 smithi070 ceph-mon[32820]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:16.778 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:16 smithi114 ceph-mon[32988]: pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 -- 172.21.15.63:0/2334330286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84103450 msgr2=0x7f7d8410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2334330286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84103450 0x7f7d8410f8b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009990 tx=0x7f7d7c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 -- 172.21.15.63:0/2334330286 shutdown_connections 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2334330286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84103450 0x7f7d8410f8b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2334330286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d84108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2334330286 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84102a90 0x7f7d84102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 -- 172.21.15.63:0/2334330286 >> 172.21.15.63:0/2334330286 conn(0x7f7d84079ba0 msgr2=0x7f7d84079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.111+0000 7f7d8a200700 1 -- 172.21.15.63:0/2334330286 shutdown_connections 2024-04-25T18:42:19.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.112+0000 7f7d8a200700 1 -- 172.21.15.63:0/2334330286 wait complete. 2024-04-25T18:42:19.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 Processor -- start 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 -- start start 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 0x7f7d841990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 0x7f7d84199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d8419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84112c20 con 0x7f7d84102a90 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84112aa0 con 0x7f7d84108a50 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.113+0000 7f7d8a200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84112da0 con 0x7f7d84103450 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d7aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 0x7f7d84199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 0x7f7d841990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d8419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d7aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 0x7f7d84199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54554/0 (socket says 172.21.15.63:54554) 2024-04-25T18:42:19.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d7aa00700 1 -- 172.21.15.63:0/2040986110 learned_addr learned my addr 172.21.15.63:0/2040986110 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:19.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 -- 172.21.15.63:0/2040986110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 msgr2=0x7f7d84199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:19.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 0x7f7d84199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 -- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 msgr2=0x7f7d841990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:19.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 0x7f7d841990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 -- 172.21.15.63:0/2040986110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8419e0b0 con 0x7f7d84108a50 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83400700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 0x7f7d841990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.114+0000 7f7d83e00700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d8419d970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0098e0 tx=0x7f7d7c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.115+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c027070 con 0x7f7d84108a50 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.115+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8419e340 con 0x7f7d84108a50 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.115+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d7c022470 con 0x7f7d84108a50 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.115+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c017650 con 0x7f7d84108a50 2024-04-25T18:42:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.115+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d841a5c20 con 0x7f7d84108a50 2024-04-25T18:42:19.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.116+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84069730 con 0x7f7d84108a50 2024-04-25T18:42:19.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.118+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7d7c02a070 con 0x7f7d84108a50 2024-04-25T18:42:19.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.119+0000 7f7d80c00700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 0x7f7d680a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:19.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.120+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f7d7c0d0c80 con 0x7f7d84108a50 2024-04-25T18:42:19.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.120+0000 7f7d83400700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 0x7f7d680a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:19.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.120+0000 7f7d83400700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 0x7f7d680a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009480 tx=0x7f7d7400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:19.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.123+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d7c05eff0 con 0x7f7d84108a50 2024-04-25T18:42:19.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: pgmap v327: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:19.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:42:19.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:42:19.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:42:19.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.357+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7d8402df80 con 0x7f7d84108a50 2024-04-25T18:42:19.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.358+0000 7f7d80c00700 1 -- 172.21.15.63:0/2040986110 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) ==== 84+0+65 (secure 0 0 0) 0x7f7d7c013070 con 0x7f7d84108a50 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout:[ 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout: "poolnum": 1, 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout: "poolname": ".mgr" 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:42:19.361 INFO:teuthology.orchestra.run.smithi063.stdout:] 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 msgr2=0x7f7d680a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 0x7f7d680a6100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009480 tx=0x7f7d7400b040 comp rx=0 tx=0).stop 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 msgr2=0x7f7d8419d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d8419d970 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0098e0 tx=0x7f7d7c0047b0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 shutdown_connections 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f7d680a3c40 0x7f7d680a6100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.361+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d84108a50 0x7f7d8419d970 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.362+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d84102a90 0x7f7d841990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.362+0000 7f7d8a200700 1 --2- 172.21.15.63:0/2040986110 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d84103450 0x7f7d84199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:19.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.362+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 >> 172.21.15.63:0/2040986110 conn(0x7f7d84079ba0 msgr2=0x7f7d841017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:19.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.362+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 shutdown_connections 2024-04-25T18:42:19.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:19.362+0000 7f7d8a200700 1 -- 172.21.15.63:0/2040986110 wait complete. 2024-04-25T18:42:20.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:19 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2040986110' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T18:42:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:19 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2040986110' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T18:42:20.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:19 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2040986110' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T18:42:20.388 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T18:42:20.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T18:42:20.884 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:21.044 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:20 smithi063 ceph-mon[28006]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:21.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:20 smithi114 ceph-mon[32988]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:21.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:20 smithi070 ceph-mon[32820]: pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.052+0000 7f0507600700 1 -- 172.21.15.63:0/524169805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 msgr2=0x7f0500108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.052+0000 7f0507600700 1 --2- 172.21.15.63:0/524169805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f0500108e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f04f40098f0 tx=0x7f04f4009c00 comp rx=0 tx=0).stop 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 -- 172.21.15.63:0/524169805 shutdown_connections 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 --2- 172.21.15.63:0/524169805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f0500108e00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 --2- 172.21.15.63:0/524169805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006b980 0x7f050010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 --2- 172.21.15.63:0/524169805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006afe0 0x7f050006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 -- 172.21.15.63:0/524169805 >> 172.21.15.63:0/524169805 conn(0x7f05000788b0 msgr2=0x7f0500078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 -- 172.21.15.63:0/524169805 shutdown_connections 2024-04-25T18:42:23.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.053+0000 7f0507600700 1 -- 172.21.15.63:0/524169805 wait complete. 2024-04-25T18:42:23.056 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:22 smithi063 ceph-mon[28006]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.054+0000 7f0507600700 1 Processor -- start 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.054+0000 7f0507600700 1 -- start start 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.054+0000 7f0507600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0507600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006b980 0x7f05001996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0507600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f050019da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0507600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0500112c10 con 0x7f050006b980 2024-04-25T18:42:23.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0507600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0500112a90 con 0x7f050006afe0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0507600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0500112d90 con 0x7f0500108a20 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57518/0 (socket says 172.21.15.63:57518) 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f04ffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006b980 0x7f05001996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 -- 172.21.15.63:0/1318353372 learned_addr learned my addr 172.21.15.63:0/1318353372 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0505600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f050019da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 -- 172.21.15.63:0/1318353372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 msgr2=0x7f050019da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f050019da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 -- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006b980 msgr2=0x7f05001996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006b980 0x7f05001996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.055+0000 7f0504c00700 1 -- 172.21.15.63:0/1318353372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050019e170 con 0x7f050006afe0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f0505600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f050019da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f0504c00700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009230 tx=0x7f04f400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4005140 con 0x7f050006afe0 2024-04-25T18:42:23.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050019e400 con 0x7f050006afe0 2024-04-25T18:42:23.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04f400b560 con 0x7f050006afe0 2024-04-25T18:42:23.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.056+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05001a5ce0 con 0x7f050006afe0 2024-04-25T18:42:23.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.057+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f4003ae0 con 0x7f050006afe0 2024-04-25T18:42:23.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.058+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0500069730 con 0x7f050006afe0 2024-04-25T18:42:23.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.060+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f04f4003c80 con 0x7f050006afe0 2024-04-25T18:42:23.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.061+0000 7f04fd600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 0x7f04e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:23.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.061+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7127+0+0 (secure 0 0 0) 0x7f04f40d0f70 con 0x7f050006afe0 2024-04-25T18:42:23.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.061+0000 7f04ffe00700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 0x7f04e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:23.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.062+0000 7f04ffe00700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 0x7f04e80a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009750 tx=0x7f04ec009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:23.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:22 smithi114 ceph-mon[32988]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:23.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.065+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04f4013030 con 0x7f050006afe0 2024-04-25T18:42:23.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:22 smithi070 ceph-mon[32820]: pgmap v329: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:23.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.297+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f050002df80 con 0x7f050006afe0 2024-04-25T18:42:23.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.814+0000 7f04fd600700 1 -- 172.21.15.63:0/1318353372 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) ==== 138+0+0 (secure 0 0 0) 0x7f04f409dde0 con 0x7f050006afe0 2024-04-25T18:42:23.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 msgr2=0x7f04e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:23.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 0x7f04e80a6100 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04ec009750 tx=0x7f04ec009310 comp rx=0 tx=0).stop 2024-04-25T18:42:23.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 msgr2=0x7f0500199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:23.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f04f4009230 tx=0x7f04f400fa70 comp rx=0 tx=0).stop 2024-04-25T18:42:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 shutdown_connections 2024-04-25T18:42:23.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f04e80a3c40 0x7f04e80a6100 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0500108a20 0x7f050019da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f050006afe0 0x7f0500199160 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.820+0000 7f0507600700 1 --2- 172.21.15.63:0/1318353372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f050006b980 0x7f05001996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.821+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 >> 172.21.15.63:0/1318353372 conn(0x7f05000788b0 msgr2=0x7f05000ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.821+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 shutdown_connections 2024-04-25T18:42:23.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:23.821+0000 7f0507600700 1 -- 172.21.15.63:0/1318353372 wait complete. 2024-04-25T18:42:23.827 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'cephfs_metadata' created 2024-04-25T18:42:24.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:23 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1318353372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:23 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:23 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1318353372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:23 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:23 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1318353372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:23 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T18:42:24.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T18:42:25.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:24 smithi114 ceph-mon[32988]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T18:42:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:24 smithi114 ceph-mon[32988]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T18:42:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:24 smithi063 ceph-mon[28006]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T18:42:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:24 smithi063 ceph-mon[28006]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T18:42:25.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:24 smithi070 ceph-mon[32820]: pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:25.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T18:42:25.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:24 smithi070 ceph-mon[32820]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T18:42:25.231 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:26.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:25 smithi063 ceph-mon[28006]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T18:42:26.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:25 smithi070 ceph-mon[32820]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T18:42:26.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:25 smithi114 ceph-mon[32988]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T18:42:26.972 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:26 smithi063 ceph-mon[28006]: pgmap v333: 33 pgs: 4 active+clean, 29 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:26.973 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:26 smithi063 ceph-mon[28006]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:26.973 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:26 smithi063 ceph-mon[28006]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T18:42:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:26 smithi070 ceph-mon[32820]: pgmap v333: 33 pgs: 4 active+clean, 29 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:26 smithi070 ceph-mon[32820]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:26 smithi070 ceph-mon[32820]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T18:42:27.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:26 smithi114 ceph-mon[32988]: pgmap v333: 33 pgs: 4 active+clean, 29 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:27.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:26 smithi114 ceph-mon[32988]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:27.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:26 smithi114 ceph-mon[32988]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T18:42:27.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 -- 172.21.15.63:0/2914089537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588077890 msgr2=0x7fc5881133e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 --2- 172.21.15.63:0/2914089537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588077890 0x7fc5881133e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc574009990 tx=0x7fc574009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 -- 172.21.15.63:0/2914089537 shutdown_connections 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 --2- 172.21.15.63:0/2914089537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588113920 0x7fc588115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 --2- 172.21.15.63:0/2914089537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588076f70 0x7fc588077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 --2- 172.21.15.63:0/2914089537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588077890 0x7fc5881133e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.496+0000 7fc58d600700 1 -- 172.21.15.63:0/2914089537 >> 172.21.15.63:0/2914089537 conn(0x7fc588100770 msgr2=0x7fc588102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.497+0000 7fc58d600700 1 -- 172.21.15.63:0/2914089537 shutdown_connections 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.497+0000 7fc58d600700 1 -- 172.21.15.63:0/2914089537 wait complete. 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 Processor -- start 2024-04-25T18:42:27.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 -- start start 2024-04-25T18:42:27.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 0x7fc5881a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:27.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:27.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:27.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc5881192f0 con 0x7fc588113920 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc588119170 con 0x7fc588076f70 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.498+0000 7fc58d600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc588119470 con 0x7fc588077890 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc587400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57320/0 (socket says 172.21.15.63:57320) 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc587400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45148/0 (socket says 172.21.15.63:45148) 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc587400700 1 -- 172.21.15.63:0/4244347392 learned_addr learned my addr 172.21.15.63:0/4244347392 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586a00700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 0x7fc5881a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 -- 172.21.15.63:0/4244347392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 msgr2=0x7fc5881a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 0x7fc5881a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 -- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 msgr2=0x7fc58819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 -- 172.21.15.63:0/4244347392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58819ffa0 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586a00700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 0x7fc5881a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc587400700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.499+0000 7fc586000700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc5740098e0 tx=0x7fc5740057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.500+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574027070 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.500+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5881a0230 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.500+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc574022470 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.500+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574017650 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.500+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5881ac570 con 0x7fc588077890 2024-04-25T18:42:27.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.501+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5881a0590 con 0x7fc588077890 2024-04-25T18:42:27.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.503+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc57402a070 con 0x7fc588077890 2024-04-25T18:42:27.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.505+0000 7fc57f400700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 0x7fc56c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:27.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.505+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7497+0+0 (secure 0 0 0) 0x7fc5740d1030 con 0x7fc588077890 2024-04-25T18:42:27.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.505+0000 7fc586a00700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 0x7fc56c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:27.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.506+0000 7fc586a00700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 0x7fc56c0a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc578005950 tx=0x7fc57800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:27.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.507+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc574031080 con 0x7fc588077890 2024-04-25T18:42:27.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.740+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc588069730 con 0x7fc588077890 2024-04-25T18:42:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.852+0000 7fc57f400700 1 -- 172.21.15.63:0/4244347392 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v89) ==== 170+0+0 (secure 0 0 0) 0x7fc574013070 con 0x7fc588077890 2024-04-25T18:42:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.860+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 msgr2=0x7fc56c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.860+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 0x7fc56c0a6180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc578005950 tx=0x7fc57800b410 comp rx=0 tx=0).stop 2024-04-25T18:42:27.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.860+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 msgr2=0x7fc5881a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:27.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.860+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc5740098e0 tx=0x7fc5740057c0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 shutdown_connections 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc56c0a3cc0 0x7fc56c0a6180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc588113920 0x7fc58819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc588076f70 0x7fc5881a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 --2- 172.21.15.63:0/4244347392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc588077890 0x7fc5881a5b30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:27.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 >> 172.21.15.63:0/4244347392 conn(0x7fc588100770 msgr2=0x7fc588102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:27.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.862+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 shutdown_connections 2024-04-25T18:42:27.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:27.863+0000 7fc58d600700 1 -- 172.21.15.63:0/4244347392 wait complete. 2024-04-25T18:42:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'cephfs_data' created 2024-04-25T18:42:28.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:27 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4244347392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:27 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:27 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4244347392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:27 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:27 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4244347392' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:27 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T18:42:28.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T18:42:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:28 smithi063 ceph-mon[28006]: pgmap v335: 33 pgs: 14 active+clean, 19 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:28 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T18:42:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:28 smithi063 ceph-mon[28006]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T18:42:29.191 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:28 smithi070 ceph-mon[32820]: pgmap v335: 33 pgs: 14 active+clean, 19 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:28 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T18:42:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:28 smithi070 ceph-mon[32820]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T18:42:29.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:28 smithi114 ceph-mon[32988]: pgmap v335: 33 pgs: 14 active+clean, 19 unknown; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:29.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:28 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T18:42:29.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:28 smithi114 ceph-mon[32988]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T18:42:30.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:29 smithi063 ceph-mon[28006]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T18:42:30.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:29 smithi070 ceph-mon[32820]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T18:42:30.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:29 smithi114 ceph-mon[32988]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T18:42:30.992 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:30 smithi063 ceph-mon[28006]: pgmap v338: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:30.992 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:30 smithi063 ceph-mon[28006]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T18:42:31.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:30 smithi070 ceph-mon[32820]: pgmap v338: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:31.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:30 smithi070 ceph-mon[32820]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T18:42:31.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:30 smithi114 ceph-mon[32988]: pgmap v338: 97 pgs: 15 creating+peering, 49 unknown, 33 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:31.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:30 smithi114 ceph-mon[32988]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T18:42:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.631+0000 7fb37d400700 1 -- 172.21.15.63:0/1586385785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb378101930 msgr2=0x7fb378101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.631+0000 7fb37d400700 1 --2- 172.21.15.63:0/1586385785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb378101930 0x7fb378101db0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb364009990 tx=0x7fb364009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 -- 172.21.15.63:0/1586385785 shutdown_connections 2024-04-25T18:42:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 --2- 172.21.15.63:0/1586385785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3781022f0 0x7fb37810e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 --2- 172.21.15.63:0/1586385785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378107980 0x7fb378107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 --2- 172.21.15.63:0/1586385785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb378101930 0x7fb378101db0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 -- 172.21.15.63:0/1586385785 >> 172.21.15.63:0/1586385785 conn(0x7fb3780788b0 msgr2=0x7fb378078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 -- 172.21.15.63:0/1586385785 shutdown_connections 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.632+0000 7fb37d400700 1 -- 172.21.15.63:0/1586385785 wait complete. 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.633+0000 7fb37d400700 1 Processor -- start 2024-04-25T18:42:31.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.633+0000 7fb37d400700 1 -- start start 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101930 0x7fb37819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 0x7fb37819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb378111ad0 con 0x7fb378107980 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb378111950 con 0x7fb378101930 2024-04-25T18:42:31.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb37d400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb378111c50 con 0x7fb3781022f0 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb377200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 0x7fb37819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb375e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.634+0000 7fb375e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57338/0 (socket says 172.21.15.63:57338) 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb376800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101930 0x7fb37819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 -- 172.21.15.63:0/955502212 learned_addr learned my addr 172.21.15.63:0/955502212 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb377200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 0x7fb37819fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45170/0 (socket says 172.21.15.63:45170) 2024-04-25T18:42:31.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 -- 172.21.15.63:0/955502212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101930 msgr2=0x7fb37819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101930 0x7fb37819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 -- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 msgr2=0x7fb37819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 0x7fb37819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 -- 172.21.15.63:0/955502212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3781a0300 con 0x7fb3781022f0 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.635+0000 7fb375e00700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3640057f0 tx=0x7fb36400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:31.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.636+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364026070 con 0x7fb3781022f0 2024-04-25T18:42:31.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.636+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb364021410 con 0x7fb3781022f0 2024-04-25T18:42:31.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.636+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364017650 con 0x7fb3781022f0 2024-04-25T18:42:31.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.636+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3781a0590 con 0x7fb3781022f0 2024-04-25T18:42:31.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.636+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3781a7e70 con 0x7fb3781022f0 2024-04-25T18:42:31.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.637+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb378103050 con 0x7fb3781022f0 2024-04-25T18:42:31.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.639+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb364029070 con 0x7fb3781022f0 2024-04-25T18:42:31.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.641+0000 7fb36f400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 0x7fb35c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:31.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.641+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7879+0+0 (secure 0 0 0) 0x7fb3640d1070 con 0x7fb3781022f0 2024-04-25T18:42:31.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.641+0000 7fb376800700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 0x7fb35c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:31.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.642+0000 7fb376800700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 0x7fb35c0a6180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb368009be0 tx=0x7fb368009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:31.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.643+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb364030080 con 0x7fb3781022f0 2024-04-25T18:42:31.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:31.922+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb378069730 con 0x7fb3781022f0 2024-04-25T18:42:32.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.894+0000 7fb36f400700 1 -- 172.21.15.63:0/955502212 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb378069730 con 0x7fb3781022f0 2024-04-25T18:42:32.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.901+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 msgr2=0x7fb35c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:32.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.901+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 0x7fb35c0a6180 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb368009be0 tx=0x7fb368009260 comp rx=0 tx=0).stop 2024-04-25T18:42:32.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.901+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 msgr2=0x7fb37819b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:32.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.901+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb3640057f0 tx=0x7fb36400bfd0 comp rx=0 tx=0).stop 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 shutdown_connections 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb35c0a3cc0 0x7fb35c0a6180 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb378107980 0x7fb37819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb378101930 0x7fb37819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 --2- 172.21.15.63:0/955502212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb3781022f0 0x7fb37819b830 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 >> 172.21.15.63:0/955502212 conn(0x7fb3780788b0 msgr2=0x7fb37806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 shutdown_connections 2024-04-25T18:42:32.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:32.902+0000 7fb37d400700 1 -- 172.21.15.63:0/955502212 wait complete. 2024-04-25T18:42:32.907 INFO:teuthology.orchestra.run.smithi063.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T18:42:33.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:32 smithi070 ceph-mon[32820]: pgmap v340: 97 pgs: 15 creating+peering, 26 unknown, 56 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:33.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:32 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/955502212' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:32 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:32 smithi070 ceph-mon[32820]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T18:42:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:32 smithi114 ceph-mon[32988]: pgmap v340: 97 pgs: 15 creating+peering, 26 unknown, 56 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:32 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/955502212' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:32 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:32 smithi114 ceph-mon[32988]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T18:42:33.344 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:32 smithi063 ceph-mon[28006]: pgmap v340: 97 pgs: 15 creating+peering, 26 unknown, 56 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:33.344 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:32 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/955502212' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.344 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:32 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T18:42:33.344 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:32 smithi063 ceph-mon[28006]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T18:42:33.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:42:34.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T18:42:34.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T18:42:34.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T18:42:34.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T18:42:34.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:34.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:active 2024-04-25T18:42:34.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:33 smithi063 ceph-mon[28006]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:34.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T18:42:34.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T18:42:34.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T18:42:34.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T18:42:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:active 2024-04-25T18:42:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:33 smithi070 ceph-mon[32820]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:34.225 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T18:42:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T18:42:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T18:42:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T18:42:34.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T18:42:34.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] up:active 2024-04-25T18:42:34.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:33 smithi114 ceph-mon[32988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:34.886 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 18:42:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: 2024-04-25T18:42:34.629+0000 7f7545024700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T18:42:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:34 smithi063 ceph-mon[28006]: pgmap v342: 97 pgs: 15 creating+peering, 82 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:34 smithi063 ceph-mon[28006]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T18:42:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:34 smithi063 ceph-mon[28006]: Cluster is now healthy 2024-04-25T18:42:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:34 smithi070 ceph-mon[32820]: pgmap v342: 97 pgs: 15 creating+peering, 82 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:34 smithi070 ceph-mon[32820]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T18:42:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:34 smithi070 ceph-mon[32820]: Cluster is now healthy 2024-04-25T18:42:35.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:34 smithi114 ceph-mon[32988]: pgmap v342: 97 pgs: 15 creating+peering, 82 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:42:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:34 smithi114 ceph-mon[32988]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T18:42:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:34 smithi114 ceph-mon[32988]: Cluster is now healthy 2024-04-25T18:42:36.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 -- 172.21.15.63:0/187729067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad806a750 msgr2=0x7f3ad806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:36.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 --2- 172.21.15.63:0/187729067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad806a750 0x7f3ad806abb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009990 tx=0x7f3ac4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 -- 172.21.15.63:0/187729067 shutdown_connections 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 --2- 172.21.15.63:0/187729067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 --2- 172.21.15.63:0/187729067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad8108a30 0x7f3ad8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 --2- 172.21.15.63:0/187729067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad806a750 0x7f3ad806abb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.478+0000 7f3adfc00700 1 -- 172.21.15.63:0/187729067 >> 172.21.15.63:0/187729067 conn(0x7f3ad80788f0 msgr2=0x7f3ad8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.479+0000 7f3adfc00700 1 -- 172.21.15.63:0/187729067 shutdown_connections 2024-04-25T18:42:36.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.479+0000 7f3adfc00700 1 -- 172.21.15.63:0/187729067 wait complete. 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 Processor -- start 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 -- start start 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 0x7f3ad8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad8108a30 0x7f3ad819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad8112c20 con 0x7f3ad806a750 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad8112aa0 con 0x7f3ad806b0f0 2024-04-25T18:42:36.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3adfc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad8112da0 con 0x7f3ad8108a30 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.480+0000 7f3add200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 0x7f3ad8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3add200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 0x7f3ad8199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45202/0 (socket says 172.21.15.63:45202) 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3add200700 1 -- 172.21.15.63:0/2958977723 learned_addr learned my addr 172.21.15.63:0/2958977723 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3addc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad8108a30 0x7f3ad819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 -- 172.21.15.63:0/2958977723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad8108a30 msgr2=0x7f3ad819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad8108a30 0x7f3ad819d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 -- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 msgr2=0x7f3ad8199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 0x7f3ad8199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 -- 172.21.15.63:0/2958977723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad819e110 con 0x7f3ad806b0f0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.481+0000 7f3ad7e00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad8199690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009990 tx=0x7f3ac40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.482+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4027070 con 0x7f3ad806b0f0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.482+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad819e3a0 con 0x7f3ad806b0f0 2024-04-25T18:42:36.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.482+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ac4022470 con 0x7f3ad806b0f0 2024-04-25T18:42:36.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.482+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4017650 con 0x7f3ad806b0f0 2024-04-25T18:42:36.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.482+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad81a5c80 con 0x7f3ad806b0f0 2024-04-25T18:42:36.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.483+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad8104170 con 0x7f3ad806b0f0 2024-04-25T18:42:36.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.485+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3ac402a070 con 0x7f3ad806b0f0 2024-04-25T18:42:36.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.486+0000 7f3ad5600700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 0x7f3ac00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:36.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.487+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f3ac40d0fb0 con 0x7f3ad806b0f0 2024-04-25T18:42:36.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.487+0000 7f3add200700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 0x7f3ac00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:36.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.487+0000 7f3add200700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 0x7f3ac00a6100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009ca0 tx=0x7f3acc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:36.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.489+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ac4013070 con 0x7f3ad806b0f0 2024-04-25T18:42:36.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.723+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ad802df80 con 0x7f3ad806b0f0 2024-04-25T18:42:36.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.725+0000 7f3ad5600700 1 -- 172.21.15.63:0/2958977723 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22172 (secure 0 0 0) 0x7f3ac409dad0 con 0x7f3ad806b0f0 2024-04-25T18:42:36.728 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:36.728 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":92,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:42:31.925599+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:42:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 msgr2=0x7f3ac00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 0x7f3ac00a6100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3acc009ca0 tx=0x7f3acc009260 comp rx=0 tx=0).stop 2024-04-25T18:42:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 msgr2=0x7f3ad8199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:36.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad8199690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009990 tx=0x7f3ac40057c0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 shutdown_connections 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3ac00a3c40 0x7f3ac00a6100 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ad8108a30 0x7f3ad819d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ad806a750 0x7f3ad8199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 --2- 172.21.15.63:0/2958977723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ad806b0f0 0x7f3ad8199690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.728+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 >> 172.21.15.63:0/2958977723 conn(0x7f3ad80788f0 msgr2=0x7f3ad80ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.729+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 shutdown_connections 2024-04-25T18:42:36.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:36.729+0000 7f3adfc00700 1 -- 172.21.15.63:0/2958977723 wait complete. 2024-04-25T18:42:37.131 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:36 smithi063 ceph-mon[28006]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 327 B/s wr, 1 op/s 2024-04-25T18:42:37.131 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:36 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2958977723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:36 smithi070 ceph-mon[32820]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 327 B/s wr, 1 op/s 2024-04-25T18:42:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:36 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2958977723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:37.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:36 smithi114 ceph-mon[32988]: pgmap v343: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 327 B/s wr, 1 op/s 2024-04-25T18:42:37.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:36 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2958977723' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:37.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:42:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:38 smithi070 ceph-mon[32820]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:39.224 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:38 smithi063 ceph-mon[28006]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:39.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:38 smithi114 ceph-mon[32988]: pgmap v344: 97 pgs: 97 active+clean; 577 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 -- 172.21.15.63:0/1133503375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c075590 msgr2=0x7fb29c10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 --2- 172.21.15.63:0/1133503375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c075590 0x7fb29c10ef00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb288009990 tx=0x7fb288009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 -- 172.21.15.63:0/1133503375 shutdown_connections 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 --2- 172.21.15.63:0/1133503375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c10f440 0x7fb29c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 --2- 172.21.15.63:0/1133503375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c074c70 0x7fb29c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 --2- 172.21.15.63:0/1133503375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c075590 0x7fb29c10ef00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 -- 172.21.15.63:0/1133503375 >> 172.21.15.63:0/1133503375 conn(0x7fb29c079ba0 msgr2=0x7fb29c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 -- 172.21.15.63:0/1133503375 shutdown_connections 2024-04-25T18:42:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.308+0000 7fb2a4600700 1 -- 172.21.15.63:0/1133503375 wait complete. 2024-04-25T18:42:40.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.309+0000 7fb2a4600700 1 Processor -- start 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 -- start start 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 0x7fb29c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c10f440 0x7fb29c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c114e50 con 0x7fb29c075590 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c114cd0 con 0x7fb29c10f440 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.310+0000 7fb2a4600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c114fd0 con 0x7fb29c074c70 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 0x7fb29c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34916/0 (socket says 172.21.15.63:34916) 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 -- 172.21.15.63:0/2357949859 learned_addr learned my addr 172.21.15.63:0/2357949859 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:40.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a2600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c10f440 0x7fb29c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 -- 172.21.15.63:0/2357949859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c10f440 msgr2=0x7fb29c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c10f440 0x7fb29c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 -- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 msgr2=0x7fb29c19b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 0x7fb29c19b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 -- 172.21.15.63:0/2357949859 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c1a0330 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1200700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 0x7fb29c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.311+0000 7fb2a1c00700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb29000ef10 tx=0x7fb29000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.312+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29000c9c0 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.312+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb290019070 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.312+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c1a05c0 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.312+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290014480 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.312+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c1a7ea0 con 0x7fb29c074c70 2024-04-25T18:42:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.313+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29c069730 con 0x7fb29c074c70 2024-04-25T18:42:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.315+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb290012050 con 0x7fb29c074c70 2024-04-25T18:42:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.316+0000 7fb29a000700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 0x7fb2840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:40.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.317+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fb2900d0200 con 0x7fb29c074c70 2024-04-25T18:42:40.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.317+0000 7fb2a1200700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 0x7fb2840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:40.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.317+0000 7fb2a1200700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 0x7fb2840a6180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb29c102570 tx=0x7fb28800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:40.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.321+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb290010040 con 0x7fb29c074c70 2024-04-25T18:42:40.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.553+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb29c02df80 con 0x7fb29c074c70 2024-04-25T18:42:40.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.555+0000 7fb29a000700 1 -- 172.21.15.63:0/2357949859 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+22172 (secure 0 0 0) 0x7fb29009cc90 con 0x7fb29c074c70 2024-04-25T18:42:40.558 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:42:40.558 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":92,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:42:31.925599+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.559+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 msgr2=0x7fb2840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.559+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 0x7fb2840a6180 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb29c102570 tx=0x7fb28800b260 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.559+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 msgr2=0x7fb29c19b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.559+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb29000ef10 tx=0x7fb29000c3a0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 shutdown_connections 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb2840a3cc0 0x7fb2840a6180 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb29c074c70 0x7fb29c19b320 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb29c10f440 0x7fb29c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 --2- 172.21.15.63:0/2357949859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb29c075590 0x7fb29c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:40.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 >> 172.21.15.63:0/2357949859 conn(0x7fb29c079ba0 msgr2=0x7fb29c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:40.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 shutdown_connections 2024-04-25T18:42:40.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:40.560+0000 7fb2a4600700 1 -- 172.21.15.63:0/2357949859 wait complete. 2024-04-25T18:42:41.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:40 smithi070 ceph-mon[32820]: pgmap v345: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-25T18:42:41.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:40 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2357949859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:41.297 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:40 smithi063 ceph-mon[28006]: pgmap v345: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-25T18:42:41.297 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:40 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2357949859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:41.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:40 smithi114 ceph-mon[32988]: pgmap v345: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-04-25T18:42:41.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:40 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2357949859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:42:41.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-04-25T18:42:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:43.052 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:42 smithi063 ceph-mon[28006]: pgmap v346: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:43.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:42 smithi070 ceph-mon[32820]: pgmap v346: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:43.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:42 smithi114 ceph-mon[32988]: pgmap v346: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 -- 172.21.15.63:0/4236704687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c074c70 msgr2=0x7f6e9c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 --2- 172.21.15.63:0/4236704687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c074c70 0x7f6e9c075050 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000b0a0 tx=0x7f6e9000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 -- 172.21.15.63:0/4236704687 shutdown_connections 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 --2- 172.21.15.63:0/4236704687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c10f440 0x7f6e9c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 --2- 172.21.15.63:0/4236704687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c074c70 0x7f6e9c075050 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 --2- 172.21.15.63:0/4236704687 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c10ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 -- 172.21.15.63:0/4236704687 >> 172.21.15.63:0/4236704687 conn(0x7f6e9c079ba0 msgr2=0x7f6e9c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 -- 172.21.15.63:0/4236704687 shutdown_connections 2024-04-25T18:42:43.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.993+0000 7f6ea4400700 1 -- 172.21.15.63:0/4236704687 wait complete. 2024-04-25T18:42:43.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.995+0000 7f6ea4400700 1 Processor -- start 2024-04-25T18:42:43.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.996+0000 7f6ea4400700 1 -- start start 2024-04-25T18:42:43.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.996+0000 7f6ea4400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 0x7f6e9c19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:43.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.996+0000 7f6ea4400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c19b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:43.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea4400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c10f440 0x7f6e9c19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:43.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea4400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c114e40 con 0x7f6e9c10f440 2024-04-25T18:42:43.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea4400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c114cc0 con 0x7f6e9c074c70 2024-04-25T18:42:43.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea4400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6e9c114fc0 con 0x7f6e9c075590 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c19b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 0x7f6e9c19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea2400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c10f440 0x7f6e9c19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c19b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34930/0 (socket says 172.21.15.63:34930) 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1000700 1 -- 172.21.15.63:0/1773501739 learned_addr learned my addr 172.21.15.63:0/1773501739 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:44.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 -- 172.21.15.63:0/1773501739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 msgr2=0x7f6e9c19b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c19b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 -- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c10f440 msgr2=0x7f6e9c19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c10f440 0x7f6e9c19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.997+0000 7f6ea1a00700 1 -- 172.21.15.63:0/1773501739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9c1a0310 con 0x7f6e9c074c70 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6ea1a00700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 0x7f6e9c19b350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000b070 tx=0x7f6e90004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90021070 con 0x7f6e9c074c70 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e90007e30 con 0x7f6e9c074c70 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e900046a0 con 0x7f6e9c074c70 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9c1a05a0 con 0x7f6e9c074c70 2024-04-25T18:42:44.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.998+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e9c1a7e80 con 0x7f6e9c074c70 2024-04-25T18:42:44.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:43.999+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e9c076360 con 0x7f6e9c074c70 2024-04-25T18:42:44.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.001+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6e90007980 con 0x7f6e9c074c70 2024-04-25T18:42:44.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.003+0000 7f6e9a000700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 0x7f6e840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:44.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.003+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7f6e9000f020 con 0x7f6e9c074c70 2024-04-25T18:42:44.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.003+0000 7f6ea1000700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 0x7f6e840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:44.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.004+0000 7f6ea1000700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 0x7f6e840a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c1111d0 tx=0x7f6e88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:44.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.007+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e9002a020 con 0x7f6e9c074c70 2024-04-25T18:42:44.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.289+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6e9c19c660 con 0x7f6e9c074c70 2024-04-25T18:42:44.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.922+0000 7f6e9a000700 1 -- 172.21.15.63:0/1773501739 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f6e9009e050 con 0x7f6e9c074c70 2024-04-25T18:42:44.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 msgr2=0x7f6e840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:44.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 0x7f6e840a6180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c1111d0 tx=0x7f6e88009260 comp rx=0 tx=0).stop 2024-04-25T18:42:44.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 msgr2=0x7f6e9c19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:44.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 0x7f6e9c19b350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e9000b070 tx=0x7f6e90004d00 comp rx=0 tx=0).stop 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 shutdown_connections 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6e840a3cc0 0x7f6e840a6180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6e9c10f440 0x7f6e9c19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e9c074c70 0x7f6e9c19b350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 --2- 172.21.15.63:0/1773501739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6e9c075590 0x7f6e9c19b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.926+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 >> 172.21.15.63:0/1773501739 conn(0x7f6e9c079ba0 msgr2=0x7f6e9c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.927+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 shutdown_connections 2024-04-25T18:42:44.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:44.927+0000 7f6ea4400700 1 -- 172.21.15.63:0/1773501739 wait complete. 2024-04-25T18:42:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:44 smithi070 ceph-mon[32820]: pgmap v347: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:44 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1773501739' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:44 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:44 smithi114 ceph-mon[32988]: pgmap v347: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:44 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1773501739' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:44 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:44 smithi063 ceph-mon[28006]: pgmap v347: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-04-25T18:42:45.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:44 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1773501739' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:44 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T18:42:45.788 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T18:42:45.788 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-04-25T18:42:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:45 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T18:42:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:45 smithi063 ceph-mon[28006]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:45 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T18:42:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:45 smithi070 ceph-mon[32820]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:46.275 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:46.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:45 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T18:42:46.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:45 smithi114 ceph-mon[32988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:46 smithi063 ceph-mon[28006]: pgmap v348: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:46 smithi070 ceph-mon[32820]: pgmap v348: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:47.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:46 smithi114 ceph-mon[32988]: pgmap v348: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-04-25T18:42:48.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 -- 172.21.15.63:0/3327933773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 msgr2=0x7fb060107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:48.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 --2- 172.21.15.63:0/3327933773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060107d70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb050009990 tx=0x7fb050009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 -- 172.21.15.63:0/3327933773 shutdown_connections 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 --2- 172.21.15.63:0/3327933773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060107d70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 --2- 172.21.15.63:0/3327933773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb06010e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 --2- 172.21.15.63:0/3327933773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb060101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 -- 172.21.15.63:0/3327933773 >> 172.21.15.63:0/3327933773 conn(0x7fb0600788b0 msgr2=0x7fb060078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 -- 172.21.15.63:0/3327933773 shutdown_connections 2024-04-25T18:42:48.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.534+0000 7fb064e00700 1 -- 172.21.15.63:0/3327933773 wait complete. 2024-04-25T18:42:48.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.535+0000 7fb064e00700 1 Processor -- start 2024-04-25T18:42:48.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.535+0000 7fb064e00700 1 -- start start 2024-04-25T18:42:48.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb060103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb060111b00 con 0x7fb060101940 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb060111980 con 0x7fb060102300 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb064e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb060111c80 con 0x7fb060107990 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb05de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb05de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44422/0 (socket says 172.21.15.63:44422) 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb05de00700 1 -- 172.21.15.63:0/1651363990 learned_addr learned my addr 172.21.15.63:0/1651363990 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb05d400700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb060103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.536+0000 7fb05e800700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 -- 172.21.15.63:0/1651363990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 msgr2=0x7fb060103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:48.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 -- 172.21.15.63:0/1651363990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 msgr2=0x7fb060103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb060103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 -- 172.21.15.63:0/1651363990 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0601a7670 con 0x7fb060101940 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05d400700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb060103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb05de00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb0500057f0 tx=0x7fb05000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:48.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050026070 con 0x7fb060101940 2024-04-25T18:42:48.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0601a7900 con 0x7fb060101940 2024-04-25T18:42:48.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb050021410 con 0x7fb060101940 2024-04-25T18:42:48.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.537+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb050017650 con 0x7fb060101940 2024-04-25T18:42:48.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.538+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0601a7de0 con 0x7fb060101940 2024-04-25T18:42:48.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.539+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb060103060 con 0x7fb060101940 2024-04-25T18:42:48.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.541+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb050029070 con 0x7fb060101940 2024-04-25T18:42:48.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.543+0000 7fb056a00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 0x7fb0440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:48.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.543+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fb0500d0f50 con 0x7fb060101940 2024-04-25T18:42:48.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.543+0000 7fb05d400700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 0x7fb0440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:48.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.544+0000 7fb05d400700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 0x7fb0440a6100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb060078db0 tx=0x7fb04c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:48.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.548+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb05005dff0 con 0x7fb060101940 2024-04-25T18:42:48.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:48.837+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fb0601a8110 con 0x7fb060101940 2024-04-25T18:42:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:48 smithi063 ceph-mon[28006]: pgmap v349: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-25T18:42:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:48 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T18:42:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:48 smithi070 ceph-mon[32820]: pgmap v349: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-25T18:42:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:48 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T18:42:49.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:48 smithi114 ceph-mon[32988]: pgmap v349: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2024-04-25T18:42:49.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:48 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-25T18:42:49.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.620+0000 7fb056a00700 1 -- 172.21.15.63:0/1651363990 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7fb050013070 con 0x7fb060101940 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.623+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 msgr2=0x7fb0440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.623+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 0x7fb0440a6100 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb060078db0 tx=0x7fb04c009260 comp rx=0 tx=0).stop 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.623+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 msgr2=0x7fb0601034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.623+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb0500057f0 tx=0x7fb05000bfd0 comp rx=0 tx=0).stop 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 shutdown_connections 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb0440a3c40 0x7fb0440a6100 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:49.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb060107990 0x7fb060103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:49.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb060101940 0x7fb0601034f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:49.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 --2- 172.21.15.63:0/1651363990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb060102300 0x7fb060103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:49.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 >> 172.21.15.63:0/1651363990 conn(0x7fb0600788b0 msgr2=0x7fb06006b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:49.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 shutdown_connections 2024-04-25T18:42:49.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:49.624+0000 7fb064e00700 1 -- 172.21.15.63:0/1651363990 wait complete. 2024-04-25T18:42:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T18:42:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T18:42:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T18:42:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T18:42:50.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:49 smithi070 ceph-mon[32820]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T18:42:50.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T18:42:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T18:42:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T18:42:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T18:42:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:49 smithi114 ceph-mon[32988]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1651363990' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T18:42:50.362 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:49 smithi063 ceph-mon[28006]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T18:42:50.444 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-04-25T18:42:50.921 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: pgmap v350: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:active 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: Cluster is now healthy 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T18:42:51.243 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:50 smithi063 ceph-mon[28006]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T18:42:51.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: pgmap v350: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:active 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: Cluster is now healthy 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T18:42:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:50 smithi114 ceph-mon[32988]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T18:42:51.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: pgmap v350: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 1 op/s 2024-04-25T18:42:51.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] up:active 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: Cluster is now healthy 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T18:42:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:50 smithi070 ceph-mon[32820]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T18:42:52.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:52 smithi063 ceph-mon[28006]: pgmap v351: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-25T18:42:52.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:52 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:active 2024-04-25T18:42:52.906 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T18:42:52.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:52 smithi070 ceph-mon[32820]: pgmap v351: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-25T18:42:52.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:52 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:active 2024-04-25T18:42:52.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T18:42:53.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:52 smithi114 ceph-mon[32988]: pgmap v351: 97 pgs: 97 active+clean; 579 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-25T18:42:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:52 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] up:active 2024-04-25T18:42:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 -- 172.21.15.63:0/4020838433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 msgr2=0x7fded01096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 --2- 172.21.15.63:0/4020838433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01096b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009920 tx=0x7fdec0009c30 comp rx=0 tx=0).stop 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 -- 172.21.15.63:0/4020838433 shutdown_connections 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 --2- 172.21.15.63:0/4020838433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01096b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 --2- 172.21.15.63:0/4020838433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 --2- 172.21.15.63:0/4020838433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 0x7fded010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 -- 172.21.15.63:0/4020838433 >> 172.21.15.63:0/4020838433 conn(0x7fded0076210 msgr2=0x7fded0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:53.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 -- 172.21.15.63:0/4020838433 shutdown_connections 2024-04-25T18:42:53.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.192+0000 7fded5600700 1 -- 172.21.15.63:0/4020838433 wait complete. 2024-04-25T18:42:53.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.193+0000 7fded5600700 1 Processor -- start 2024-04-25T18:42:53.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 -- start start 2024-04-25T18:42:53.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:53.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 0x7fded01a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:53.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:53.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fded0119270 con 0x7fded0109c80 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fded01190f0 con 0x7fded010e3b0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fded5600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fded01193f0 con 0x7fded01092d0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fdecea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fdecea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58664/0 (socket says 172.21.15.63:58664) 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fdece000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 0x7fded01a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fdecea00700 1 -- 172.21.15.63:0/3374668262 learned_addr learned my addr 172.21.15.63:0/3374668262 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.194+0000 7fdecf400700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 -- 172.21.15.63:0/3374668262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 msgr2=0x7fded019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 -- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 msgr2=0x7fded01a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 0x7fded01a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 -- 172.21.15.63:0/3374668262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded019fd50 con 0x7fded01092d0 2024-04-25T18:42:53.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecf400700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdecea00700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009230 tx=0x7fdec000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec0027070 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdec00055f0 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded019ffe0 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.195+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec0004930 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.196+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded01ac300 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.197+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded0069730 con 0x7fded01092d0 2024-04-25T18:42:53.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.199+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdec002a070 con 0x7fded01092d0 2024-04-25T18:42:53.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.201+0000 7fdec7400700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 0x7fdeb40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:53.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.201+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 7979+0+0 (secure 0 0 0) 0x7fdec00d0fe0 con 0x7fded01092d0 2024-04-25T18:42:53.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.201+0000 7fdece000700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 0x7fdeb40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:53.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.202+0000 7fdece000700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 0x7fdeb40a6180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fded0077160 tx=0x7fdebc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:53.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.202+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdec0031020 con 0x7fded01092d0 2024-04-25T18:42:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.482+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fded002df80 con 0x7fded01092d0 2024-04-25T18:42:53.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.624+0000 7fdec7400700 1 -- 172.21.15.63:0/3374668262 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7fdec0013030 con 0x7fded01092d0 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.628+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 msgr2=0x7fdeb40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.628+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 0x7fdeb40a6180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fded0077160 tx=0x7fdebc009310 comp rx=0 tx=0).stop 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.628+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 msgr2=0x7fded01a5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.628+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009230 tx=0x7fdec000fa00 comp rx=0 tx=0).stop 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 shutdown_connections 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdeb40a3cc0 0x7fdeb40a6180 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fded01092d0 0x7fded01a5430 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fded010e3b0 0x7fded019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 --2- 172.21.15.63:0/3374668262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fded0109c80 0x7fded01a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 >> 172.21.15.63:0/3374668262 conn(0x7fded0076210 msgr2=0x7fded0077e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.629+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 shutdown_connections 2024-04-25T18:42:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:53.630+0000 7fded5600700 1 -- 172.21.15.63:0/3374668262 wait complete. 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3374668262' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:53.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T18:42:53.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:53 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:42:53.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3374668262' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T18:42:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:53 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3374668262' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T18:42:54.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:53 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:42:54.540 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: pgmap v352: 97 pgs: 97 active+clean; 580 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 4 op/s 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-25T18:42:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T18:42:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T18:42:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:54 smithi063 ceph-mon[28006]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T18:42:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: pgmap v352: 97 pgs: 97 active+clean; 580 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 4 op/s 2024-04-25T18:42:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T18:42:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:54 smithi070 ceph-mon[32820]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T18:42:55.017 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: pgmap v352: 97 pgs: 97 active+clean; 580 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 4 op/s 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 6]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4, 7, 11]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 7]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 6]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-04-25T18:42:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:54 smithi114 ceph-mon[32988]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T18:42:55.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:55 smithi063 ceph-mon[28006]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T18:42:55.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:55 smithi070 ceph-mon[32820]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T18:42:56.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:55 smithi114 ceph-mon[32988]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T18:42:56.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:56 smithi063 ceph-mon[28006]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-25T18:42:56.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:56 smithi070 ceph-mon[32820]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-25T18:42:57.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:56 smithi114 ceph-mon[32988]: pgmap v355: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 3.0 KiB/s wr, 10 op/s 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 -- 172.21.15.63:0/1729321875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38075790 msgr2=0x7f5e3810ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/1729321875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38075790 0x7f5e3810ec60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5e24009990 tx=0x7f5e24009ca0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 -- 172.21.15.63:0/1729321875 shutdown_connections 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/1729321875 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e38075250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/1729321875 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e3810f1a0 0x7f5e38111610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/1729321875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e38075790 0x7f5e3810ec60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 -- 172.21.15.63:0/1729321875 >> 172.21.15.63:0/1729321875 conn(0x7f5e380fe450 msgr2=0x7f5e38100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 -- 172.21.15.63:0/1729321875 shutdown_connections 2024-04-25T18:42:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.142+0000 7f5e3cc00700 1 -- 172.21.15.63:0/1729321875 wait complete. 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.143+0000 7f5e3cc00700 1 Processor -- start 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 -- start start 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e3819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e38075790 0x7f5e3819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 0x7f5e3819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5e38114d70 con 0x7f5e3810f1a0 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e38114bf0 con 0x7f5e38075790 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e3cc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e38114ef0 con 0x7f5e38074e70 2024-04-25T18:42:57.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 0x7f5e3819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e37e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 0x7f5e3819fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44460/0 (socket says 172.21.15.63:44460) 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e37e00700 1 -- 172.21.15.63:0/438546535 learned_addr learned my addr 172.21.15.63:0/438546535 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e37400700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e3819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.144+0000 7f5e36a00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e38075790 0x7f5e3819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 -- 172.21.15.63:0/438546535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e38075790 msgr2=0x7f5e3819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e38075790 0x7f5e3819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 -- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 msgr2=0x7f5e3819fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 0x7f5e3819fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 -- 172.21.15.63:0/438546535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e381a02d0 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.145+0000 7f5e37400700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e3819b2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9f0 tx=0x7f5e2800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.146+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800c9c0 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.146+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e2800cb60 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.146+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e381a05c0 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.146+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28018680 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.146+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e381a7e40 con 0x7f5e38074e70 2024-04-25T18:42:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.147+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e38069730 con 0x7f5e38074e70 2024-04-25T18:42:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.149+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5e28010430 con 0x7f5e38074e70 2024-04-25T18:42:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.150+0000 7f5e2fe00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 0x7f5e1c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:42:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.151+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f5e280d0570 con 0x7f5e38074e70 2024-04-25T18:42:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.151+0000 7f5e36a00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 0x7f5e1c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:42:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.151+0000 7f5e36a00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 0x7f5e1c0a6100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5e38110640 tx=0x7f5e2400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:42:57.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.155+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e2809cef0 con 0x7f5e38074e70 2024-04-25T18:42:57.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.438+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f5e3802df80 con 0x7f5e38074e70 2024-04-25T18:42:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.699+0000 7f5e2fe00700 1 -- 172.21.15.63:0/438546535 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f5e2809c7f0 con 0x7f5e38074e70 2024-04-25T18:42:57.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:57 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/438546535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:57.701 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:57 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 msgr2=0x7f5e1c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 0x7f5e1c0a6100 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5e38110640 tx=0x7f5e2400b260 comp rx=0 tx=0).stop 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 msgr2=0x7f5e3819b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e3819b2c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9f0 tx=0x7f5e2800ed00 comp rx=0 tx=0).stop 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 shutdown_connections 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e1c0a3c40 0x7f5e1c0a6100 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e38074e70 0x7f5e3819b2c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e3810f1a0 0x7f5e3819fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 --2- 172.21.15.63:0/438546535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e38075790 0x7f5e3819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:42:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 >> 172.21.15.63:0/438546535 conn(0x7f5e380fe450 msgr2=0x7f5e380ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:42:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.703+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 shutdown_connections 2024-04-25T18:42:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:42:57.704+0000 7f5e3cc00700 1 -- 172.21.15.63:0/438546535 wait complete. 2024-04-25T18:42:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:57 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/438546535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:57 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:57 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/438546535' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:57 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T18:42:58.495 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T18:42:58.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-04-25T18:42:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:58 smithi070 ceph-mon[32820]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.0 KiB/s wr, 11 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T18:42:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:58 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T18:42:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:42:58 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:42:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:42:59.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:58 smithi114 ceph-mon[32988]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.0 KiB/s wr, 11 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T18:42:59.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:58 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T18:42:59.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:42:58 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:42:59.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:58 smithi063 ceph-mon[28006]: pgmap v356: 97 pgs: 1 peering, 96 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 3.0 KiB/s wr, 11 op/s; 13 B/s, 1 objects/s recovering 2024-04-25T18:42:59.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:58 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T18:42:59.118 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:42:58 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:43:01.042 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:00 smithi063 ceph-mon[28006]: pgmap v357: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.0 KiB/s wr, 15 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:00 smithi114 ceph-mon[32988]: pgmap v357: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.0 KiB/s wr, 15 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:01.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:00 smithi070 ceph-mon[32820]: pgmap v357: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 3.0 KiB/s wr, 15 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 -- 172.21.15.63:0/1207080821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 msgr2=0x7fd25806b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 --2- 172.21.15.63:0/1207080821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25806b430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd244009990 tx=0x7fd244009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 -- 172.21.15.63:0/1207080821 shutdown_connections 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 --2- 172.21.15.63:0/1207080821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd258108a20 0x7fd258108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 --2- 172.21.15.63:0/1207080821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd25806b970 0x7fd25810f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 --2- 172.21.15.63:0/1207080821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25806b430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 -- 172.21.15.63:0/1207080821 >> 172.21.15.63:0/1207080821 conn(0x7fd258078780 msgr2=0x7fd258078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 -- 172.21.15.63:0/1207080821 shutdown_connections 2024-04-25T18:43:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.218+0000 7fd25e400700 1 -- 172.21.15.63:0/1207080821 wait complete. 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.219+0000 7fd25e400700 1 Processor -- start 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 -- start start 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25819b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd25806b970 0x7fd25819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd258112bf0 con 0x7fd258108a20 2024-04-25T18:43:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd258112a70 con 0x7fd25806afb0 2024-04-25T18:43:01.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25e400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd258112d70 con 0x7fd25806b970 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.220+0000 7fd25da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd257e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd25806b970 0x7fd25819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25da00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37130/0 (socket says 172.21.15.63:37130) 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25da00700 1 -- 172.21.15.63:0/2386062669 learned_addr learned my addr 172.21.15.63:0/2386062669 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25819b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 -- 172.21.15.63:0/2386062669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd25806b970 msgr2=0x7fd25819b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd25806b970 0x7fd25819b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 -- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 msgr2=0x7fd25819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 -- 172.21.15.63:0/2386062669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2581a01d0 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25da00700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.221+0000 7fd25d000700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25819b1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e9f0 tx=0x7fd24c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.222+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c004220 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.222+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd24c0043c0 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.222+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24c005020 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.222+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2581a04c0 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.222+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2581a7d40 con 0x7fd25806afb0 2024-04-25T18:43:01.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.223+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd258069730 con 0x7fd25806afb0 2024-04-25T18:43:01.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.225+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd24c004750 con 0x7fd25806afb0 2024-04-25T18:43:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.226+0000 7fd255600700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 0x7fd2400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.227+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7fd24c0d0310 con 0x7fd25806afb0 2024-04-25T18:43:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.227+0000 7fd257e00700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 0x7fd2400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.227+0000 7fd257e00700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 0x7fd2400a6180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd258078d10 tx=0x7fd244005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.231+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd24c09cc90 con 0x7fd25806afb0 2024-04-25T18:43:01.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.499+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd258062e80 con 0x7fd2400a3cc0 2024-04-25T18:43:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.536+0000 7fd255600700 1 -- 172.21.15.63:0/2386062669 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd258062e80 con 0x7fd2400a3cc0 2024-04-25T18:43:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 msgr2=0x7fd2400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 0x7fd2400a6180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd258078d10 tx=0x7fd244005fd0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 msgr2=0x7fd25819b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25819b1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd24c00e9f0 tx=0x7fd24c00ed00 comp rx=0 tx=0).stop 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 shutdown_connections 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd2400a3cc0 0x7fd2400a6180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd258108a20 0x7fd25819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd25806afb0 0x7fd25819b1c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 --2- 172.21.15.63:0/2386062669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd25806b970 0x7fd25819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 >> 172.21.15.63:0/2386062669 conn(0x7fd258078780 msgr2=0x7fd25810dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 shutdown_connections 2024-04-25T18:43:01.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:01.539+0000 7fd25e400700 1 -- 172.21.15.63:0/2386062669 wait complete. 2024-04-25T18:43:01.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:43:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:43:01.513+0000 7f1d26a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:43:01.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:43:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:43:01.513+0000 7f1d26a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:43:01.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:43:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:43:01.513+0000 7f1d26a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:43:01.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:43:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:43:01.513+0000 7f1d26a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:43:01.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:43:01 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:43:01.513+0000 7f1d26a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:43:02.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T18:43:02.834 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:02.989 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:02 smithi063 ceph-mon[28006]: pgmap v358: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:02.989 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:02 smithi063 ceph-mon[28006]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:03.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:02 smithi114 ceph-mon[32988]: pgmap v358: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:02 smithi114 ceph-mon[32988]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:03.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:02 smithi070 ceph-mon[32820]: pgmap v358: 97 pgs: 97 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.5 KiB/s wr, 8 op/s; 86 B/s, 1 objects/s recovering 2024-04-25T18:43:03.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:02 smithi070 ceph-mon[32820]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:04 smithi063 ceph-mon[28006]: pgmap v359: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 73 B/s, 1 objects/s recovering 2024-04-25T18:43:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:04 smithi063 ceph-mon[28006]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 -- 172.21.15.63:0/495227432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1840108a20 msgr2=0x7f1840108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 --2- 172.21.15.63:0/495227432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1840108a20 0x7f1840108e00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1830009990 tx=0x7f1830009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 -- 172.21.15.63:0/495227432 shutdown_connections 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 --2- 172.21.15.63:0/495227432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006b980 0x7f184010f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 --2- 172.21.15.63:0/495227432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1840108a20 0x7f1840108e00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 --2- 172.21.15.63:0/495227432 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f184006afe0 0x7f184006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 -- 172.21.15.63:0/495227432 >> 172.21.15.63:0/495227432 conn(0x7f18400788b0 msgr2=0x7f1840078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:05.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.060+0000 7f1845600700 1 -- 172.21.15.63:0/495227432 shutdown_connections 2024-04-25T18:43:05.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.061+0000 7f1845600700 1 -- 172.21.15.63:0/495227432 wait complete. 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 Processor -- start 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 -- start start 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 0x7f1840101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1840108a20 0x7f1840102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1840112bf0 con 0x7f184006b980 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1840112a70 con 0x7f184006afe0 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.062+0000 7f1845600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1840112d70 con 0x7f1840108a20 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37154/0 (socket says 172.21.15.63:37154) 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 -- 172.21.15.63:0/3731691780 learned_addr learned my addr 172.21.15.63:0/3731691780 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183ea00700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 0x7f1840101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183f400700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1840108a20 0x7f1840102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 -- 172.21.15.63:0/3731691780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1840108a20 msgr2=0x7f1840102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1840108a20 0x7f1840102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 -- 172.21.15.63:0/3731691780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 msgr2=0x7f1840101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 0x7f1840101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183e000700 1 -- 172.21.15.63:0/3731691780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18401abad0 con 0x7f184006b980 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.063+0000 7f183ea00700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 0x7f1840101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f183e000700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f182c00e9a0 tx=0x7f182c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c00ca10 con 0x7f184006b980 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f182c00cbb0 con 0x7f184006b980 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c018680 con 0x7f184006b980 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18401abd60 con 0x7f184006b980 2024-04-25T18:43:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.064+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18401ac190 con 0x7f184006b980 2024-04-25T18:43:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:04 smithi114 ceph-mon[32988]: pgmap v359: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 73 B/s, 1 objects/s recovering 2024-04-25T18:43:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:04 smithi114 ceph-mon[32988]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-25T18:43:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.065+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1840104160 con 0x7f184006b980 2024-04-25T18:43:05.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.068+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f182c010430 con 0x7f184006b980 2024-04-25T18:43:05.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.069+0000 7f1837400700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 0x7f18240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:05.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.069+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f182c0d0790 con 0x7f184006b980 2024-04-25T18:43:05.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.069+0000 7f183ea00700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 0x7f18240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:05.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.070+0000 7f183ea00700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 0x7f18240a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f183000b2e0 tx=0x7f183001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:05.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.075+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f182c09cfb0 con 0x7f184006b980 2024-04-25T18:43:05.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:04 smithi070 ceph-mon[32820]: pgmap v359: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 73 B/s, 1 objects/s recovering 2024-04-25T18:43:05.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:04 smithi070 ceph-mon[32820]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-25T18:43:05.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.347+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1840062e80 con 0x7f18240a3cd0 2024-04-25T18:43:05.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.392+0000 7f1837400700 1 -- 172.21.15.63:0/3731691780 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1840062e80 con 0x7f18240a3cd0 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.395+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 msgr2=0x7f18240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.395+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 0x7f18240a6190 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f183000b2e0 tx=0x7f183001a040 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.395+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 msgr2=0x7f1840101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.395+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f182c00e9a0 tx=0x7f182c00ecb0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 shutdown_connections 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f18240a3cd0 0x7f18240a6190 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1840108a20 0x7f1840102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f184006afe0 0x7f1840101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 --2- 172.21.15.63:0/3731691780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f184006b980 0x7f1840101f40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:05.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 >> 172.21.15.63:0/3731691780 conn(0x7f18400788b0 msgr2=0x7f18400ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:05.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 shutdown_connections 2024-04-25T18:43:05.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:05.396+0000 7f1845600700 1 -- 172.21.15.63:0/3731691780 wait complete. 2024-04-25T18:43:06.160 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T18:43:06.635 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:07.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:06 smithi063 ceph-mon[28006]: pgmap v360: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 66 B/s, 1 objects/s recovering 2024-04-25T18:43:07.019 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:06 smithi063 ceph-mon[28006]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:06 smithi114 ceph-mon[32988]: pgmap v360: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 66 B/s, 1 objects/s recovering 2024-04-25T18:43:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:06 smithi114 ceph-mon[32988]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:07.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:06 smithi070 ceph-mon[32820]: pgmap v360: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.1 KiB/s wr, 8 op/s; 66 B/s, 1 objects/s recovering 2024-04-25T18:43:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:06 smithi070 ceph-mon[32820]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 -- 172.21.15.63:0/3896270751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0109fc0 msgr2=0x7f4bc0111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/3896270751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0109fc0 0x7f4bc0111b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4bb8009990 tx=0x7f4bb8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 -- 172.21.15.63:0/3896270751 shutdown_connections 2024-04-25T18:43:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/3896270751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0109fc0 0x7f4bc0111b90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/3896270751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0104ee0 0x7f4bc01052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/3896270751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc01098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 -- 172.21.15.63:0/3896270751 >> 172.21.15.63:0/3896270751 conn(0x7f4bc0100770 msgr2=0x7f4bc0102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 -- 172.21.15.63:0/3896270751 shutdown_connections 2024-04-25T18:43:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.911+0000 7f4bc4e00700 1 -- 172.21.15.63:0/3896270751 wait complete. 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.912+0000 7f4bc4e00700 1 Processor -- start 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.912+0000 7f4bc4e00700 1 -- start start 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 0x7f4bc019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0109fc0 0x7f4bc019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0114ec0 con 0x7f4bc0109fc0 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0114d40 con 0x7f4bc0104ee0 2024-04-25T18:43:08.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bc4e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc0115040 con 0x7f4bc0105890 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 0x7f4bc019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bbd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bbe800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0109fc0 0x7f4bc019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bbde00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 0x7f4bc019b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59334/0 (socket says 172.21.15.63:59334) 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.913+0000 7f4bbde00700 1 -- 172.21.15.63:0/313570821 learned_addr learned my addr 172.21.15.63:0/313570821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 -- 172.21.15.63:0/313570821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 msgr2=0x7f4bc019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 0x7f4bc019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 -- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0109fc0 msgr2=0x7f4bc019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0109fc0 0x7f4bc019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 -- 172.21.15.63:0/313570821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc01a0300 con 0x7f4bc0105890 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bbd400700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc019b830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800ef90 tx=0x7f4ba800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:08.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80092e0 con 0x7f4bc0105890 2024-04-25T18:43:08.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc01a0590 con 0x7f4bc0105890 2024-04-25T18:43:08.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba8004bb0 con 0x7f4bc0105890 2024-04-25T18:43:08.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8004020 con 0x7f4bc0105890 2024-04-25T18:43:08.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.914+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc01a7e70 con 0x7f4bc0105890 2024-04-25T18:43:08.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.916+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc0069730 con 0x7f4bc0105890 2024-04-25T18:43:08.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.918+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4ba801b030 con 0x7f4bc0105890 2024-04-25T18:43:08.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.919+0000 7f4bb6a00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 0x7f4ba00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:08.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.919+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f4ba80d0480 con 0x7f4bc0105890 2024-04-25T18:43:08.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.919+0000 7f4bbde00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 0x7f4ba00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:08.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.920+0000 7f4bbde00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 0x7f4ba00a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0009d20 tx=0x7f4bb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:08.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:08.922+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ba8022020 con 0x7f4bc0105890 2024-04-25T18:43:09.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:08 smithi114 ceph-mon[32988]: pgmap v361: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-04-25T18:43:09.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:08 smithi063 ceph-mon[28006]: pgmap v361: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-04-25T18:43:09.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.191+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4bc0101940 con 0x7f4ba00a3cd0 2024-04-25T18:43:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:08 smithi070 ceph-mon[32820]: pgmap v361: 97 pgs: 97 active+clean; 588 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 6 op/s; 57 B/s, 1 objects/s recovering 2024-04-25T18:43:09.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.202+0000 7f4bb6a00700 1 -- 172.21.15.63:0/313570821 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4bc0101940 con 0x7f4ba00a3cd0 2024-04-25T18:43:09.204 INFO:teuthology.orchestra.run.smithi063.stdout:/volumes/qa/sv_0/65ae298c-de57-47a3-bd32-b7446366487b 2024-04-25T18:43:09.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.205+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 msgr2=0x7f4ba00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:09.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.205+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 0x7f4ba00a6190 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0009d20 tx=0x7f4bb0009260 comp rx=0 tx=0).stop 2024-04-25T18:43:09.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.205+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 msgr2=0x7f4bc019b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.205+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc019b830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800ef90 tx=0x7f4ba800c3a0 comp rx=0 tx=0).stop 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 shutdown_connections 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f4ba00a3cd0 0x7f4ba00a6190 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bc0109fc0 0x7f4bc019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bc0104ee0 0x7f4bc019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 --2- 172.21.15.63:0/313570821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4bc0105890 0x7f4bc019b830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:09.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 >> 172.21.15.63:0/313570821 conn(0x7f4bc0100770 msgr2=0x7f4bc0101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:09.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 shutdown_connections 2024-04-25T18:43:09.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:09.206+0000 7f4bc4e00700 1 -- 172.21.15.63:0/313570821 wait complete. 2024-04-25T18:43:10.009 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T18:43:10.520 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:11.005 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:10 smithi063 ceph-mon[28006]: pgmap v362: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T18:43:11.005 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:10 smithi063 ceph-mon[28006]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:11.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:10 smithi114 ceph-mon[32988]: pgmap v362: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T18:43:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:10 smithi114 ceph-mon[32988]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:11.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:10 smithi070 ceph-mon[32820]: pgmap v362: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.2 KiB/s wr, 9 op/s; 48 B/s, 0 objects/s recovering 2024-04-25T18:43:11.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:10 smithi070 ceph-mon[32820]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 -- 172.21.15.63:0/2335610849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214104ec0 msgr2=0x7f12141052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 --2- 172.21.15.63:0/2335610849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214104ec0 0x7f12141052a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12080099c0 tx=0x7f1208009cd0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 -- 172.21.15.63:0/2335610849 shutdown_connections 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 --2- 172.21.15.63:0/2335610849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214104ec0 0x7f12141052a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 --2- 172.21.15.63:0/2335610849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 0x7f1214111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 --2- 172.21.15.63:0/2335610849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214105870 0x7f12141098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 -- 172.21.15.63:0/2335610849 >> 172.21.15.63:0/2335610849 conn(0x7f1214100750 msgr2=0x7f1214102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:12.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 -- 172.21.15.63:0/2335610849 shutdown_connections 2024-04-25T18:43:12.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.791+0000 7f121c000700 1 -- 172.21.15.63:0/2335610849 wait complete. 2024-04-25T18:43:12.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.792+0000 7f121c000700 1 Processor -- start 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 -- start start 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214105870 0x7f121419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 0x7f121419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1214114e50 con 0x7f1214104ec0 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1214114cd0 con 0x7f1214109fa0 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121c000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1214114fd0 con 0x7f1214105870 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f1219600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f1219600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46280/0 (socket says 172.21.15.63:46280) 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f1219600700 1 -- 172.21.15.63:0/2347885552 learned_addr learned my addr 172.21.15.63:0/2347885552 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:12.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f1218c00700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214105870 0x7f121419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.793+0000 7f121a000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 0x7f121419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 -- 172.21.15.63:0/2347885552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214105870 msgr2=0x7f121419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214105870 0x7f121419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 -- 172.21.15.63:0/2347885552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 msgr2=0x7f121419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 0x7f121419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 -- 172.21.15.63:0/2347885552 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12141a0330 con 0x7f1214104ec0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1219600700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1208006010 tx=0x7f1208005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208026070 con 0x7f1214104ec0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12141a05c0 con 0x7f1214104ec0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1208005680 con 0x7f1214104ec0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.794+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120800fb20 con 0x7f1214104ec0 2024-04-25T18:43:12.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.795+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12141a7ea0 con 0x7f1214104ec0 2024-04-25T18:43:12.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.796+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1214069730 con 0x7f1214104ec0 2024-04-25T18:43:12.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.798+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1208029070 con 0x7f1214104ec0 2024-04-25T18:43:12.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.799+0000 7f1212000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 0x7f11fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:12.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.799+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f12080d0d00 con 0x7f1214104ec0 2024-04-25T18:43:12.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.799+0000 7f1218c00700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 0x7f11fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:12.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.800+0000 7f1218c00700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 0x7f11fc0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1214101b70 tx=0x7f1200009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:12.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:12.804+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f120805dff0 con 0x7f1214104ec0 2024-04-25T18:43:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:12 smithi063 ceph-mon[28006]: pgmap v363: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-25T18:43:13.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:12 smithi114 ceph-mon[32988]: pgmap v363: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-25T18:43:13.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.069+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1214062e80 con 0x7f11fc0a3cd0 2024-04-25T18:43:13.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.105+0000 7f1212000700 1 -- 172.21.15.63:0/2347885552 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1214062e80 con 0x7f11fc0a3cd0 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.108+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 msgr2=0x7f11fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.108+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 0x7f11fc0a6190 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1214101b70 tx=0x7f1200009260 comp rx=0 tx=0).stop 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.108+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 msgr2=0x7f121419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.108+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1208006010 tx=0x7f1208005280 comp rx=0 tx=0).stop 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 shutdown_connections 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f11fc0a3cd0 0x7f11fc0a6190 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1214109fa0 0x7f121419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:13.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1214104ec0 0x7f121419b320 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:13.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 --2- 172.21.15.63:0/2347885552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1214105870 0x7f121419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:13.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 >> 172.21.15.63:0/2347885552 conn(0x7f1214100750 msgr2=0x7f1214109130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:13.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 shutdown_connections 2024-04-25T18:43:13.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:13.109+0000 7f121c000700 1 -- 172.21.15.63:0/2347885552 wait complete. 2024-04-25T18:43:13.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:12 smithi070 ceph-mon[32820]: pgmap v363: 97 pgs: 97 active+clean; 602 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2024-04-25T18:43:13.947 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T18:43:14.416 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:14.831 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:14 smithi063 ceph-mon[28006]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:14.831 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:14 smithi063 ceph-mon[28006]: pgmap v364: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-25T18:43:15.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:14 smithi114 ceph-mon[32988]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:14 smithi114 ceph-mon[32988]: pgmap v364: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-25T18:43:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:14 smithi070 ceph-mon[32820]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:14 smithi070 ceph-mon[32820]: pgmap v364: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2024-04-25T18:43:16.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 -- 172.21.15.63:0/2021452171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 msgr2=0x7f79c81052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 --2- 172.21.15.63:0/2021452171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c81052c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f79b80099c0 tx=0x7f79b8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 -- 172.21.15.63:0/2021452171 shutdown_connections 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 --2- 172.21.15.63:0/2021452171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c81052c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 --2- 172.21.15.63:0/2021452171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c8111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 --2- 172.21.15.63:0/2021452171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.904+0000 7f79cd000700 1 -- 172.21.15.63:0/2021452171 >> 172.21.15.63:0/2021452171 conn(0x7f79c8100770 msgr2=0x7f79c8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.905+0000 7f79cd000700 1 -- 172.21.15.63:0/2021452171 shutdown_connections 2024-04-25T18:43:16.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.905+0000 7f79cd000700 1 -- 172.21.15.63:0/2021452171 wait complete. 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.906+0000 7f79cd000700 1 Processor -- start 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.906+0000 7f79cd000700 1 -- start start 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c810eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c810f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8114ed0 con 0x7f79c8109fc0 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8114d50 con 0x7f79c8105890 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79cd000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8115050 con 0x7f79c8104ee0 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c5e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c5e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59510/0 (socket says 172.21.15.63:59510) 2024-04-25T18:43:16.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c5e00700 1 -- 172.21.15.63:0/1510729926 learned_addr learned my addr 172.21.15.63:0/1510729926 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c5400700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c810eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c6800700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c810f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.907+0000 7f79c5e00700 1 -- 172.21.15.63:0/1510729926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 msgr2=0x7f79c810eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5e00700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c810eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5e00700 1 -- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 msgr2=0x7f79c810f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5e00700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c810f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5e00700 1 -- 172.21.15.63:0/1510729926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c81abc10 con 0x7f79c8104ee0 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5e00700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006010 tx=0x7f79b80055f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c6800700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:43:16.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79c5400700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8026070 con 0x7f79c8104ee0 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c81abde0 con 0x7f79c8104ee0 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79b800f460 con 0x7f79c8104ee0 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.908+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8009e60 con 0x7f79c8104ee0 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.909+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c81ac1d0 con 0x7f79c8104ee0 2024-04-25T18:43:16.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.910+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c81063c0 con 0x7f79c8104ee0 2024-04-25T18:43:16.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.912+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f79b8029070 con 0x7f79c8104ee0 2024-04-25T18:43:16.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.913+0000 7f79bea00700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 0x7f79ac0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:16.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.914+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f79b80d0c60 con 0x7f79c8104ee0 2024-04-25T18:43:16.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.914+0000 7f79c5400700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 0x7f79ac0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:16.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.914+0000 7f79c5400700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 0x7f79ac0a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79c81102a0 tx=0x7f79b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:16.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:16.918+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79b8013070 con 0x7f79c8104ee0 2024-04-25T18:43:17.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:16 smithi114 ceph-mon[32988]: pgmap v365: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:43:17.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:16 smithi063 ceph-mon[28006]: pgmap v365: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:43:17.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.190+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f79c8062e80 con 0x7f79ac0a3d50 2024-04-25T18:43:17.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.198+0000 7f79bea00700 1 -- 172.21.15.63:0/1510729926 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f79c8062e80 con 0x7f79ac0a3d50 2024-04-25T18:43:17.199 INFO:teuthology.orchestra.run.smithi063.stdout:/volumes/qa/sv_1/fb5cabf5-5965-43c6-9259-1d222383970f 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 msgr2=0x7f79ac0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 0x7f79ac0a6210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79c81102a0 tx=0x7f79b4009260 comp rx=0 tx=0).stop 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 msgr2=0x7f79c8111a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006010 tx=0x7f79b80055f0 comp rx=0 tx=0).stop 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 shutdown_connections 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f79ac0a3d50 0x7f79ac0a6210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79c8104ee0 0x7f79c8111a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:17.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79c8109fc0 0x7f79c810f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:17.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.200+0000 7f79cd000700 1 --2- 172.21.15.63:0/1510729926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79c8105890 0x7f79c810eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:17.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.201+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 >> 172.21.15.63:0/1510729926 conn(0x7f79c8100770 msgr2=0x7f79c8102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:17.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.201+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 shutdown_connections 2024-04-25T18:43:17.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:17.201+0000 7f79cd000700 1 -- 172.21.15.63:0/1510729926 wait complete. 2024-04-25T18:43:17.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:16 smithi070 ceph-mon[32820]: pgmap v365: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:43:17.957 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:43:18.432 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:18 smithi063 ceph-mon[28006]: pgmap v366: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-25T18:43:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:18 smithi063 ceph-mon[28006]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:18 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:19.041 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:18 smithi070 ceph-mon[32820]: pgmap v366: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-25T18:43:19.041 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:18 smithi070 ceph-mon[32820]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:19.041 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:18 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:19.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:18 smithi114 ceph-mon[32988]: pgmap v366: 97 pgs: 97 active+clean; 608 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-25T18:43:19.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:18 smithi114 ceph-mon[32988]: from='client.24646 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:43:19.055 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:18 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:43:20.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 -- 172.21.15.63:0/1499181205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb801034b0 msgr2=0x7fdb8010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:20.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1499181205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb801034b0 0x7fdb8010f920 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800b0d0 tx=0x7fdb7800b3e0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 -- 172.21.15.63:0/1499181205 shutdown_connections 2024-04-25T18:43:20.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1499181205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb801034b0 0x7fdb8010f920 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1499181205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80107940 0x7fdb80107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1499181205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb80102af0 0x7fdb80102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 -- 172.21.15.63:0/1499181205 >> 172.21.15.63:0/1499181205 conn(0x7fdb80079ba0 msgr2=0x7fdb80079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:20.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 -- 172.21.15.63:0/1499181205 shutdown_connections 2024-04-25T18:43:20.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.721+0000 7fdb86c00700 1 -- 172.21.15.63:0/1499181205 wait complete. 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.723+0000 7fdb86c00700 1 Processor -- start 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.723+0000 7fdb86c00700 1 -- start start 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 0x7fdb80075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 0x7fdb800757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80112c60 con 0x7fdb80107940 2024-04-25T18:43:20.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80112ae0 con 0x7fdb80102af0 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb86c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80112de0 con 0x7fdb801034b0 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb7fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb7fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59262/0 (socket says 172.21.15.63:59262) 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb7fe00700 1 -- 172.21.15.63:0/1454185383 learned_addr learned my addr 172.21.15.63:0/1454185383 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb84c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 0x7fdb800757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:20.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.724+0000 7fdb7f400700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 0x7fdb80075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 -- 172.21.15.63:0/1454185383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 msgr2=0x7fdb80075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 0x7fdb80075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 -- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 msgr2=0x7fdb800757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 0x7fdb800757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 -- 172.21.15.63:0/1454185383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb801a75b0 con 0x7fdb80102af0 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb84c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 0x7fdb800757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:20.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7f400700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 0x7fdb80075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7fe00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000eb30 tx=0x7fdb7000ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000cbd0 con 0x7fdb80102af0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.725+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb801a78a0 con 0x7fdb80102af0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.726+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb801a7e30 con 0x7fdb80102af0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.726+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb700044d0 con 0x7fdb80102af0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.726+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70003e40 con 0x7fdb80102af0 2024-04-25T18:43:20.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.727+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb60005330 con 0x7fdb80102af0 2024-04-25T18:43:20.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.728+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdb70004020 con 0x7fdb80102af0 2024-04-25T18:43:20.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.730+0000 7fdb7cc00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 0x7fdb640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:20.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.730+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7fdb700d0740 con 0x7fdb80102af0 2024-04-25T18:43:20.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.730+0000 7fdb7f400700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 0x7fdb640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:20.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.731+0000 7fdb7f400700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 0x7fdb640a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdb8007a130 tx=0x7fdb6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:20.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:20.734+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb7009cf60 con 0x7fdb80102af0 2024-04-25T18:43:20.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:20 smithi063 ceph-mon[28006]: pgmap v367: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-04-25T18:43:20.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:20.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:20.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:20.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:20 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.026+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb600058f0 con 0x7fdb80102af0 2024-04-25T18:43:21.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.027+0000 7fdb7cc00700 1 -- 172.21.15.63:0/1454185383 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7fdb7009c860 con 0x7fdb80102af0 2024-04-25T18:43:21.029 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:21.029 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:43:21.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 msgr2=0x7fdb640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:21.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 0x7fdb640a6190 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdb8007a130 tx=0x7fdb6c009260 comp rx=0 tx=0).stop 2024-04-25T18:43:21.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 msgr2=0x7fdb80074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:21.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000eb30 tx=0x7fdb7000ee40 comp rx=0 tx=0).stop 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 shutdown_connections 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fdb640a3cd0 0x7fdb640a6190 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb80107940 0x7fdb800757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb80102af0 0x7fdb80074d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 --2- 172.21.15.63:0/1454185383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdb801034b0 0x7fdb80075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:21.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.032+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 >> 172.21.15.63:0/1454185383 conn(0x7fdb80079ba0 msgr2=0x7fdb80100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:21.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.033+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 shutdown_connections 2024-04-25T18:43:21.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:21.033+0000 7fdb86c00700 1 -- 172.21.15.63:0/1454185383 wait complete. 2024-04-25T18:43:21.035 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:43:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:20 smithi114 ceph-mon[32988]: pgmap v367: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-04-25T18:43:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:20 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:20 smithi070 ceph-mon[32820]: pgmap v367: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-04-25T18:43:21.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:21.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:20 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:22.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:21 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1454185383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:43:22.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:21 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1454185383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:43:22.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:21 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1454185383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:43:22.296 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:43:22.750 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:23.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:22 smithi114 ceph-mon[32988]: pgmap v368: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-25T18:43:23.099 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:22 smithi063 ceph-mon[28006]: pgmap v368: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-25T18:43:23.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:22 smithi070 ceph-mon[32820]: pgmap v368: 97 pgs: 97 active+clean; 630 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-25T18:43:24.461 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:24 smithi063 ceph-mon[28006]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-04-25T18:43:24.461 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:24 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.462 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:24 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:24 smithi114 ceph-mon[32988]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-04-25T18:43:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:24 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:24 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:24 smithi070 ceph-mon[32820]: pgmap v369: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-04-25T18:43:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:24 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:24 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:24.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.995+0000 7f9902c00700 1 -- 172.21.15.63:0/3675994039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc105870 msgr2=0x7f98fc1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:24.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.995+0000 7f9902c00700 1 --2- 172.21.15.63:0/3675994039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc105870 0x7f98fc1098c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98e80099c0 tx=0x7f98e8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:43:24.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 -- 172.21.15.63:0/3675994039 shutdown_connections 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 --2- 172.21.15.63:0/3675994039 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc104ec0 0x7f98fc1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 --2- 172.21.15.63:0/3675994039 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc109fa0 0x7f98fc111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 --2- 172.21.15.63:0/3675994039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc105870 0x7f98fc1098c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 -- 172.21.15.63:0/3675994039 >> 172.21.15.63:0/3675994039 conn(0x7f98fc100750 msgr2=0x7f98fc102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 -- 172.21.15.63:0/3675994039 shutdown_connections 2024-04-25T18:43:24.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.996+0000 7f9902c00700 1 -- 172.21.15.63:0/3675994039 wait complete. 2024-04-25T18:43:24.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.997+0000 7f9902c00700 1 Processor -- start 2024-04-25T18:43:24.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.997+0000 7f9902c00700 1 -- start start 2024-04-25T18:43:24.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:24.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 0x7f98fc19b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:24.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc109fa0 0x7f98fc19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc114e80 con 0x7f98fc109fa0 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc114d00 con 0x7f98fc104ec0 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9902c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f98fc115000 con 0x7f98fc105870 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f98fbe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f98fbe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59266/0 (socket says 172.21.15.63:59266) 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f98fbe00700 1 -- 172.21.15.63:0/1437876284 learned_addr learned my addr 172.21.15.63:0/1437876284 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f9900c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc109fa0 0x7f98fc19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:25.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.998+0000 7f98fb400700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 0x7f98fc19b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 -- 172.21.15.63:0/1437876284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 msgr2=0x7f98fc19b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 0x7f98fc19b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 -- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc109fa0 msgr2=0x7f98fc19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc109fa0 0x7f98fc19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 -- 172.21.15.63:0/1437876284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98fc1a0370 con 0x7f98fc104ec0 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fb400700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 0x7f98fc19b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98fbe00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00e9c0 tx=0x7f98ec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00cbd0 con 0x7f98fc104ec0 2024-04-25T18:43:25.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec00ee50 con 0x7f98fc104ec0 2024-04-25T18:43:25.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98fc1a0660 con 0x7f98fc104ec0 2024-04-25T18:43:25.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:24.999+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec005550 con 0x7f98fc104ec0 2024-04-25T18:43:25.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.000+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98fc1a7ee0 con 0x7f98fc104ec0 2024-04-25T18:43:25.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.001+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98fc1063a0 con 0x7f98fc104ec0 2024-04-25T18:43:25.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.003+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f98ec004750 con 0x7f98fc104ec0 2024-04-25T18:43:25.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.004+0000 7f98f8c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 0x7f98e00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:25.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.004+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f98ec0d0840 con 0x7f98fc104ec0 2024-04-25T18:43:25.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.004+0000 7f98fb400700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 0x7f98e00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:25.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.005+0000 7f98fb400700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 0x7f98e00a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:25.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.009+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98ec09d060 con 0x7f98fc104ec0 2024-04-25T18:43:25.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.248+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98fc19c6f0 con 0x7f98fc104ec0 2024-04-25T18:43:25.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.250+0000 7f98f8c00700 1 -- 172.21.15.63:0/1437876284 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22643 (secure 0 0 0) 0x7f98fc19c6f0 con 0x7f98fc104ec0 2024-04-25T18:43:25.252 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:25.252 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":94,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:42:54.660943+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 msgr2=0x7f98e00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 0x7f98e00a6190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8005fb0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 msgr2=0x7f98fc19b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00e9c0 tx=0x7f98ec00ecd0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 shutdown_connections 2024-04-25T18:43:25.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f98e00a3cd0 0x7f98e00a6190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98fc109fa0 0x7f98fc19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f98fc104ec0 0x7f98fc19b3b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 --2- 172.21.15.63:0/1437876284 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f98fc105870 0x7f98fc19b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 >> 172.21.15.63:0/1437876284 conn(0x7f98fc100750 msgr2=0x7f98fc102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 shutdown_connections 2024-04-25T18:43:25.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:25.255+0000 7f9902c00700 1 -- 172.21.15.63:0/1437876284 wait complete. 2024-04-25T18:43:25.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:25.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:25.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:25.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:43:25.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:43:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:43:26.101 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T18:43:26.101 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:43:26.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:26 smithi063 ceph-mon[28006]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-25T18:43:26.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:26 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1437876284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:26.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:26 smithi114 ceph-mon[32988]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-25T18:43:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:26 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1437876284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:26.575 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:26 smithi070 ceph-mon[32820]: pgmap v370: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 7 op/s 2024-04-25T18:43:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:26 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1437876284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:28.364 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:28 smithi063 ceph-mon[28006]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-25T18:43:28.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:28 smithi114 ceph-mon[32988]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-25T18:43:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:28 smithi070 ceph-mon[32820]: pgmap v371: 97 pgs: 97 active+clean; 635 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-04-25T18:43:28.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 -- 172.21.15.63:0/550658422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 msgr2=0x7f0be81096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:28.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 --2- 172.21.15.63:0/550658422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81096b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80099c0 tx=0x7f0bd8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 -- 172.21.15.63:0/550658422 shutdown_connections 2024-04-25T18:43:28.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 --2- 172.21.15.63:0/550658422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81096b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 --2- 172.21.15.63:0/550658422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 0x7f0be8115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 --2- 172.21.15.63:0/550658422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 -- 172.21.15.63:0/550658422 >> 172.21.15.63:0/550658422 conn(0x7f0be8076210 msgr2=0x7f0be8107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:28.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 -- 172.21.15.63:0/550658422 shutdown_connections 2024-04-25T18:43:28.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.904+0000 7f0bed000700 1 -- 172.21.15.63:0/550658422 wait complete. 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.906+0000 7f0bed000700 1 Processor -- start 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.906+0000 7f0bed000700 1 -- start start 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.906+0000 7f0bed000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.906+0000 7f0bed000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.906+0000 7f0bed000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 0x7f0be819f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bed000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0be8119290 con 0x7f0be810e3b0 2024-04-25T18:43:28.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bed000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0be8119110 con 0x7f0be8109c80 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bed000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0be8119410 con 0x7f0be81092d0 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bdfe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 0x7f0be819f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bdfe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36086/0 (socket says 172.21.15.63:36086) 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bdfe00700 1 -- 172.21.15.63:0/3905554107 learned_addr learned my addr 172.21.15.63:0/3905554107 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 -- 172.21.15.63:0/3905554107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 msgr2=0x7f0be81a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:28.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 -- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 msgr2=0x7f0be819f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 0x7f0be819f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0be5e00700 1 -- 172.21.15.63:0/3905554107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be819fd10 con 0x7f0be81092d0 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.907+0000 7f0bdfe00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0be5e00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81a53f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098e0 tx=0x7f0bd8005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8026070 con 0x7f0be81092d0 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0bd8003c70 con 0x7f0be81092d0 2024-04-25T18:43:28.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800f970 con 0x7f0be81092d0 2024-04-25T18:43:28.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be819ffa0 con 0x7f0be81092d0 2024-04-25T18:43:28.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.908+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be81ac2c0 con 0x7f0be81092d0 2024-04-25T18:43:28.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.909+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0be810a7b0 con 0x7f0be81092d0 2024-04-25T18:43:28.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.911+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0bd8029070 con 0x7f0be81092d0 2024-04-25T18:43:28.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.913+0000 7f0bdea00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 0x7f0bcc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:28.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.913+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f0bd80d0ca0 con 0x7f0be81092d0 2024-04-25T18:43:28.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.913+0000 7f0bdfe00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 0x7f0bcc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:28.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.913+0000 7f0bdfe00700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 0x7f0bcc0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0be80782a0 tx=0x7f0bd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:28.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:28.915+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bd8013070 con 0x7f0be81092d0 2024-04-25T18:43:29.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.141+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0be81a0ae0 con 0x7f0bcc0a3cd0 2024-04-25T18:43:29.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.151+0000 7f0bdea00700 1 -- 172.21.15.63:0/3905554107 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258736 (secure 0 0 0) 0x7f0be81a0ae0 con 0x7f0bcc0a3cd0 2024-04-25T18:43:29.159 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 msgr2=0x7f0bcc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 0x7f0bcc0a6190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0be80782a0 tx=0x7f0bd0009260 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 msgr2=0x7f0be81a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81a53f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098e0 tx=0x7f0bd8005280 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 shutdown_connections 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f0bcc0a3cd0 0x7f0bcc0a6190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0be81092d0 0x7f0be81a53f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be810e3b0 0x7f0be819f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 --2- 172.21.15.63:0/3905554107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0be8109c80 0x7f0be81a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 >> 172.21.15.63:0/3905554107 conn(0x7f0be8076210 msgr2=0x7f0be810cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 shutdown_connections 2024-04-25T18:43:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:29.159+0000 7f0bed000700 1 -- 172.21.15.63:0/3905554107 wait complete. 2024-04-25T18:43:29.163 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:43:30.035 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-04-25T18:43:27.142743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":650108,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":340,"num_read_kb":281,"num_write":344,"num_write_kb":3045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388196,"kb_used_data":14072,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553628,"statfs":{"total":1151940427776,"available":1151542915072,"internally_reserved":0,"allocated":14409728,"data_stored":7914582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27422,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":58,"num_write":17,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004971"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675644+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:42:54.675644+0000","last_peered":"2024-04-25T18:42:54.675644+0000","last_clean":"2024-04-25T18:42:54.675644+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:42:54.675644+0000","last_undegraded":"2024-04-25T18:42:54.675644+0000","last_fullsized":"2024-04-25T18:42:54.675644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685047+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:42:54.685047+0000","last_peered":"2024-04-25T18:42:54.685047+0000","last_clean":"2024-04-25T18:42:54.685047+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:42:54.685047+0000","last_undegraded":"2024-04-25T18:42:54.685047+0000","last_fullsized":"2024-04-25T18:42:54.685047+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684922+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:42:54.684922+0000","last_peered":"2024-04-25T18:42:54.684922+0000","last_clean":"2024-04-25T18:42:54.684922+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:42:54.684922+0000","last_undegraded":"2024-04-25T18:42:54.684922+0000","last_fullsized":"2024-04-25T18:42:54.684922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112496+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:42:55.112496+0000","last_peered":"2024-04-25T18:42:55.112496+0000","last_clean":"2024-04-25T18:42:55.112496+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:42:55.112496+0000","last_undegraded":"2024-04-25T18:42:55.112496+0000","last_fullsized":"2024-04-25T18:42:55.112496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677379+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:42:54.677379+0000","last_peered":"2024-04-25T18:42:54.677379+0000","last_clean":"2024-04-25T18:42:54.677379+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:42:54.677379+0000","last_undegraded":"2024-04-25T18:42:54.677379+0000","last_fullsized":"2024-04-25T18:42:54.677379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673197+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:42:54.673197+0000","last_peered":"2024-04-25T18:42:54.673197+0000","last_clean":"2024-04-25T18:42:54.673197+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:42:54.673197+0000","last_undegraded":"2024-04-25T18:42:54.673197+0000","last_fullsized":"2024-04-25T18:42:54.673197+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678909+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:42:54.678909+0000","last_peered":"2024-04-25T18:42:54.678909+0000","last_clean":"2024-04-25T18:42:54.678909+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:42:54.678909+0000","last_undegraded":"2024-04-25T18:42:54.678909+0000","last_fullsized":"2024-04-25T18:42:54.678909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674648+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:42:54.674648+0000","last_peered":"2024-04-25T18:42:54.674648+0000","last_clean":"2024-04-25T18:42:54.674648+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:42:54.674648+0000","last_undegraded":"2024-04-25T18:42:54.674648+0000","last_fullsized":"2024-04-25T18:42:54.674648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677434+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:42:54.677434+0000","last_peered":"2024-04-25T18:42:54.677434+0000","last_clean":"2024-04-25T18:42:54.677434+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:42:54.677434+0000","last_undegraded":"2024-04-25T18:42:54.677434+0000","last_fullsized":"2024-04-25T18:42:54.677434+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673511+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:42:54.673511+0000","last_peered":"2024-04-25T18:42:54.673511+0000","last_clean":"2024-04-25T18:42:54.673511+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:42:54.673511+0000","last_undegraded":"2024-04-25T18:42:54.673511+0000","last_fullsized":"2024-04-25T18:42:54.673511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679526+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:42:54.679526+0000","last_peered":"2024-04-25T18:42:54.679526+0000","last_clean":"2024-04-25T18:42:54.679526+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:42:54.679526+0000","last_undegraded":"2024-04-25T18:42:54.679526+0000","last_fullsized":"2024-04-25T18:42:54.679526+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112449+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:42:55.112449+0000","last_peered":"2024-04-25T18:42:55.112449+0000","last_clean":"2024-04-25T18:42:55.112449+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:42:55.112449+0000","last_undegraded":"2024-04-25T18:42:55.112449+0000","last_fullsized":"2024-04-25T18:42:55.112449+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382799+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:42:55.382799+0000","last_peered":"2024-04-25T18:42:55.382799+0000","last_clean":"2024-04-25T18:42:55.382799+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:42:55.382799+0000","last_undegraded":"2024-04-25T18:42:55.382799+0000","last_fullsized":"2024-04-25T18:42:55.382799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684494+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:42:54.684494+0000","last_peered":"2024-04-25T18:42:54.684494+0000","last_clean":"2024-04-25T18:42:54.684494+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:42:54.684494+0000","last_undegraded":"2024-04-25T18:42:54.684494+0000","last_fullsized":"2024-04-25T18:42:54.684494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.683167+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:42:54.683167+0000","last_peered":"2024-04-25T18:42:54.683167+0000","last_clean":"2024-04-25T18:42:54.683167+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:42:54.683167+0000","last_undegraded":"2024-04-25T18:42:54.683167+0000","last_fullsized":"2024-04-25T18:42:54.683167+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676652+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:42:54.676652+0000","last_peered":"2024-04-25T18:42:54.676652+0000","last_clean":"2024-04-25T18:42:54.676652+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:42:54.676652+0000","last_undegraded":"2024-04-25T18:42:54.676652+0000","last_fullsized":"2024-04-25T18:42:54.676652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675880+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:42:54.675880+0000","last_peered":"2024-04-25T18:42:54.675880+0000","last_clean":"2024-04-25T18:42:54.675880+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:42:54.675880+0000","last_undegraded":"2024-04-25T18:42:54.675880+0000","last_fullsized":"2024-04-25T18:42:54.675880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680693+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:42:54.680693+0000","last_peered":"2024-04-25T18:42:54.680693+0000","last_clean":"2024-04-25T18:42:54.680693+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:42:54.680693+0000","last_undegraded":"2024-04-25T18:42:54.680693+0000","last_fullsized":"2024-04-25T18:42:54.680693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673892+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:42:54.673892+0000","last_peered":"2024-04-25T18:42:54.673892+0000","last_clean":"2024-04-25T18:42:54.673892+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:42:54.673892+0000","last_undegraded":"2024-04-25T18:42:54.673892+0000","last_fullsized":"2024-04-25T18:42:54.673892+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.699790+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:42:54.699790+0000","last_peered":"2024-04-25T18:42:54.699790+0000","last_clean":"2024-04-25T18:42:54.699790+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:42:54.699790+0000","last_undegraded":"2024-04-25T18:42:54.699790+0000","last_fullsized":"2024-04-25T18:42:54.699790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676165+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:42:54.676165+0000","last_peered":"2024-04-25T18:42:54.676165+0000","last_clean":"2024-04-25T18:42:54.676165+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:42:54.676165+0000","last_undegraded":"2024-04-25T18:42:54.676165+0000","last_fullsized":"2024-04-25T18:42:54.676165+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676461+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:42:54.676461+0000","last_peered":"2024-04-25T18:42:54.676461+0000","last_clean":"2024-04-25T18:42:54.676461+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:42:54.676461+0000","last_undegraded":"2024-04-25T18:42:54.676461+0000","last_fullsized":"2024-04-25T18:42:54.676461+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685320+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:42:54.685320+0000","last_peered":"2024-04-25T18:42:54.685320+0000","last_clean":"2024-04-25T18:42:54.685320+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:42:54.685320+0000","last_undegraded":"2024-04-25T18:42:54.685320+0000","last_fullsized":"2024-04-25T18:42:54.685320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676702+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:42:54.676702+0000","last_peered":"2024-04-25T18:42:54.676702+0000","last_clean":"2024-04-25T18:42:54.676702+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:42:54.676702+0000","last_undegraded":"2024-04-25T18:42:54.676702+0000","last_fullsized":"2024-04-25T18:42:54.676702+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676905+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:42:54.676905+0000","last_peered":"2024-04-25T18:42:54.676905+0000","last_clean":"2024-04-25T18:42:54.676905+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:42:54.676905+0000","last_undegraded":"2024-04-25T18:42:54.676905+0000","last_fullsized":"2024-04-25T18:42:54.676905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677652+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:42:54.677652+0000","last_peered":"2024-04-25T18:42:54.677652+0000","last_clean":"2024-04-25T18:42:54.677652+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:42:54.677652+0000","last_undegraded":"2024-04-25T18:42:54.677652+0000","last_fullsized":"2024-04-25T18:42:54.677652+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.681497+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:42:54.681497+0000","last_peered":"2024-04-25T18:42:54.681497+0000","last_clean":"2024-04-25T18:42:54.681497+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:42:54.681497+0000","last_undegraded":"2024-04-25T18:42:54.681497+0000","last_fullsized":"2024-04-25T18:42:54.681497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:22.671243+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:43:22.671243+0000","last_peered":"2024-04-25T18:43:22.671243+0000","last_clean":"2024-04-25T18:43:22.671243+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:43:22.671243+0000","last_undegraded":"2024-04-25T18:43:22.671243+0000","last_fullsized":"2024-04-25T18:43:22.671243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.384178+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:42:55.384178+0000","last_peered":"2024-04-25T18:42:55.384178+0000","last_clean":"2024-04-25T18:42:55.384178+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:42:55.384178+0000","last_undegraded":"2024-04-25T18:42:55.384178+0000","last_fullsized":"2024-04-25T18:42:55.384178+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673085+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:42:54.673085+0000","last_peered":"2024-04-25T18:42:54.673085+0000","last_clean":"2024-04-25T18:42:54.673085+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:42:54.673085+0000","last_undegraded":"2024-04-25T18:42:54.673085+0000","last_fullsized":"2024-04-25T18:42:54.673085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677767+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:42:54.677767+0000","last_peered":"2024-04-25T18:42:54.677767+0000","last_clean":"2024-04-25T18:42:54.677767+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:42:54.677767+0000","last_undegraded":"2024-04-25T18:42:54.677767+0000","last_fullsized":"2024-04-25T18:42:54.677767+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382290+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:42:55.382290+0000","last_peered":"2024-04-25T18:42:55.382290+0000","last_clean":"2024-04-25T18:42:55.382290+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:42:55.382290+0000","last_undegraded":"2024-04-25T18:42:55.382290+0000","last_fullsized":"2024-04-25T18:42:55.382290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674960+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:42:54.674960+0000","last_peered":"2024-04-25T18:42:54.674960+0000","last_clean":"2024-04-25T18:42:54.674960+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:42:54.674960+0000","last_undegraded":"2024-04-25T18:42:54.674960+0000","last_fullsized":"2024-04-25T18:42:54.674960+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677070+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:42:54.677070+0000","last_peered":"2024-04-25T18:42:54.677070+0000","last_clean":"2024-04-25T18:42:54.677070+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:42:54.677070+0000","last_undegraded":"2024-04-25T18:42:54.677070+0000","last_fullsized":"2024-04-25T18:42:54.677070+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680411+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:42:54.680411+0000","last_peered":"2024-04-25T18:42:54.680411+0000","last_clean":"2024-04-25T18:42:54.680411+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:42:54.680411+0000","last_undegraded":"2024-04-25T18:42:54.680411+0000","last_fullsized":"2024-04-25T18:42:54.680411+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:21.669160+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:43:21.669160+0000","last_peered":"2024-04-25T18:43:21.669160+0000","last_clean":"2024-04-25T18:43:21.669160+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:43:21.669160+0000","last_undegraded":"2024-04-25T18:43:21.669160+0000","last_fullsized":"2024-04-25T18:43:21.669160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677259+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:42:54.677259+0000","last_peered":"2024-04-25T18:42:54.677259+0000","last_clean":"2024-04-25T18:42:54.677259+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:42:54.677259+0000","last_undegraded":"2024-04-25T18:42:54.677259+0000","last_fullsized":"2024-04-25T18:42:54.677259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.118076+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:42:55.118076+0000","last_peered":"2024-04-25T18:42:55.118076+0000","last_clean":"2024-04-25T18:42:55.118076+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:42:55.118076+0000","last_undegraded":"2024-04-25T18:42:55.118076+0000","last_fullsized":"2024-04-25T18:42:55.118076+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685396+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:42:54.685396+0000","last_peered":"2024-04-25T18:42:54.685396+0000","last_clean":"2024-04-25T18:42:54.685396+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:42:54.685396+0000","last_undegraded":"2024-04-25T18:42:54.685396+0000","last_fullsized":"2024-04-25T18:42:54.685396+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678632+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:42:54.678632+0000","last_peered":"2024-04-25T18:42:54.678632+0000","last_clean":"2024-04-25T18:42:54.678632+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:42:54.678632+0000","last_undegraded":"2024-04-25T18:42:54.678632+0000","last_fullsized":"2024-04-25T18:42:54.678632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680465+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:42:54.680465+0000","last_peered":"2024-04-25T18:42:54.680465+0000","last_clean":"2024-04-25T18:42:54.680465+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:42:54.680465+0000","last_undegraded":"2024-04-25T18:42:54.680465+0000","last_fullsized":"2024-04-25T18:42:54.680465+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'28","reported_seq":122,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:22.671421+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:43:22.671421+0000","last_peered":"2024-04-25T18:43:22.671421+0000","last_clean":"2024-04-25T18:43:22.671421+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:43:22.671421+0000","last_undegraded":"2024-04-25T18:43:22.671421+0000","last_fullsized":"2024-04-25T18:43:22.671421+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55505,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":68,"num_read_kb":87,"num_write":29,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676420+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:42:54.676420+0000","last_peered":"2024-04-25T18:42:54.676420+0000","last_clean":"2024-04-25T18:42:54.676420+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:42:54.676420+0000","last_undegraded":"2024-04-25T18:42:54.676420+0000","last_fullsized":"2024-04-25T18:42:54.676420+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673719+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:42:54.673719+0000","last_peered":"2024-04-25T18:42:54.673719+0000","last_clean":"2024-04-25T18:42:54.673719+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:42:54.673719+0000","last_undegraded":"2024-04-25T18:42:54.673719+0000","last_fullsized":"2024-04-25T18:42:54.673719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382354+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:42:55.382354+0000","last_peered":"2024-04-25T18:42:55.382354+0000","last_clean":"2024-04-25T18:42:55.382354+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:42:55.382354+0000","last_undegraded":"2024-04-25T18:42:55.382354+0000","last_fullsized":"2024-04-25T18:42:55.382354+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677488+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:42:54.677488+0000","last_peered":"2024-04-25T18:42:54.677488+0000","last_clean":"2024-04-25T18:42:54.677488+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:42:54.677488+0000","last_undegraded":"2024-04-25T18:42:54.677488+0000","last_fullsized":"2024-04-25T18:42:54.677488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679419+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:42:54.679419+0000","last_peered":"2024-04-25T18:42:54.679419+0000","last_clean":"2024-04-25T18:42:54.679419+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:42:54.679419+0000","last_undegraded":"2024-04-25T18:42:54.679419+0000","last_fullsized":"2024-04-25T18:42:54.679419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'4","reported_seq":63,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:21.685428+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:43:21.685428+0000","last_peered":"2024-04-25T18:43:21.685428+0000","last_clean":"2024-04-25T18:43:21.685428+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:43:21.685428+0000","last_undegraded":"2024-04-25T18:43:21.685428+0000","last_fullsized":"2024-04-25T18:43:21.685428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678855+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:42:54.678855+0000","last_peered":"2024-04-25T18:42:54.678855+0000","last_clean":"2024-04-25T18:42:54.678855+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:42:54.678855+0000","last_undegraded":"2024-04-25T18:42:54.678855+0000","last_fullsized":"2024-04-25T18:42:54.678855+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.144779+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:42:55.144779+0000","last_peered":"2024-04-25T18:42:55.144779+0000","last_clean":"2024-04-25T18:42:55.144779+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:42:55.144779+0000","last_undegraded":"2024-04-25T18:42:55.144779+0000","last_fullsized":"2024-04-25T18:42:55.144779+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677070+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:42:54.677070+0000","last_peered":"2024-04-25T18:42:54.677070+0000","last_clean":"2024-04-25T18:42:54.677070+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:42:54.677070+0000","last_undegraded":"2024-04-25T18:42:54.677070+0000","last_fullsized":"2024-04-25T18:42:54.677070+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675999+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:42:54.675999+0000","last_peered":"2024-04-25T18:42:54.675999+0000","last_clean":"2024-04-25T18:42:54.675999+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:42:54.675999+0000","last_undegraded":"2024-04-25T18:42:54.675999+0000","last_fullsized":"2024-04-25T18:42:54.675999+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676579+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:42:54.676579+0000","last_peered":"2024-04-25T18:42:54.676579+0000","last_clean":"2024-04-25T18:42:54.676579+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:42:54.676579+0000","last_undegraded":"2024-04-25T18:42:54.676579+0000","last_fullsized":"2024-04-25T18:42:54.676579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680841+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:42:54.680841+0000","last_peered":"2024-04-25T18:42:54.680841+0000","last_clean":"2024-04-25T18:42:54.680841+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:42:54.680841+0000","last_undegraded":"2024-04-25T18:42:54.680841+0000","last_fullsized":"2024-04-25T18:42:54.680841+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673405+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:42:54.673405+0000","last_peered":"2024-04-25T18:42:54.673405+0000","last_clean":"2024-04-25T18:42:54.673405+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:42:54.673405+0000","last_undegraded":"2024-04-25T18:42:54.673405+0000","last_fullsized":"2024-04-25T18:42:54.673405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679258+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:42:54.679258+0000","last_peered":"2024-04-25T18:42:54.679258+0000","last_clean":"2024-04-25T18:42:54.679258+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:42:54.679258+0000","last_undegraded":"2024-04-25T18:42:54.679258+0000","last_fullsized":"2024-04-25T18:42:54.679258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673123+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:42:54.673123+0000","last_peered":"2024-04-25T18:42:54.673123+0000","last_clean":"2024-04-25T18:42:54.673123+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:42:54.673123+0000","last_undegraded":"2024-04-25T18:42:54.673123+0000","last_fullsized":"2024-04-25T18:42:54.673123+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677448+0000","last_change":"2024-04-25T18:42:28.876775+0000","last_active":"2024-04-25T18:42:54.677448+0000","last_peered":"2024-04-25T18:42:54.677448+0000","last_clean":"2024-04-25T18:42:54.677448+0000","last_became_active":"2024-04-25T18:42:28.876377+0000","last_became_peered":"2024-04-25T18:42:28.876377+0000","last_unstale":"2024-04-25T18:42:54.677448+0000","last_undegraded":"2024-04-25T18:42:54.677448+0000","last_fullsized":"2024-04-25T18:42:54.677448+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:34.927698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'4","reported_seq":60,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:21.684775+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:43:21.684775+0000","last_peered":"2024-04-25T18:43:21.684775+0000","last_clean":"2024-04-25T18:43:21.684775+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:43:21.684775+0000","last_undegraded":"2024-04-25T18:43:21.684775+0000","last_fullsized":"2024-04-25T18:43:21.684775+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676068+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:42:54.676068+0000","last_peered":"2024-04-25T18:42:54.676068+0000","last_clean":"2024-04-25T18:42:54.676068+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:42:54.676068+0000","last_undegraded":"2024-04-25T18:42:54.676068+0000","last_fullsized":"2024-04-25T18:42:54.676068+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112350+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:42:55.112350+0000","last_peered":"2024-04-25T18:42:55.112350+0000","last_clean":"2024-04-25T18:42:55.112350+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:42:55.112350+0000","last_undegraded":"2024-04-25T18:42:55.112350+0000","last_fullsized":"2024-04-25T18:42:55.112350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":362,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679157+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:42:54.679157+0000","last_peered":"2024-04-25T18:42:54.679157+0000","last_clean":"2024-04-25T18:42:54.679157+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:42:54.679157+0000","last_undegraded":"2024-04-25T18:42:54.679157+0000","last_fullsized":"2024-04-25T18:42:54.679157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.381856+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:42:55.381856+0000","last_peered":"2024-04-25T18:42:55.381856+0000","last_clean":"2024-04-25T18:42:55.381856+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:42:55.381856+0000","last_undegraded":"2024-04-25T18:42:55.381856+0000","last_fullsized":"2024-04-25T18:42:55.381856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679322+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:42:54.679322+0000","last_peered":"2024-04-25T18:42:54.679322+0000","last_clean":"2024-04-25T18:42:54.679322+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:42:54.679322+0000","last_undegraded":"2024-04-25T18:42:54.679322+0000","last_fullsized":"2024-04-25T18:42:54.679322+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676523+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:42:54.676523+0000","last_peered":"2024-04-25T18:42:54.676523+0000","last_clean":"2024-04-25T18:42:54.676523+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:42:54.676523+0000","last_undegraded":"2024-04-25T18:42:54.676523+0000","last_fullsized":"2024-04-25T18:42:54.676523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382850+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:42:55.382850+0000","last_peered":"2024-04-25T18:42:55.382850+0000","last_clean":"2024-04-25T18:42:55.382850+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:42:55.382850+0000","last_undegraded":"2024-04-25T18:42:55.382850+0000","last_fullsized":"2024-04-25T18:42:55.382850+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'10","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:14.908454+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:43:14.908454+0000","last_peered":"2024-04-25T18:43:14.908454+0000","last_clean":"2024-04-25T18:43:14.908454+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:43:14.908454+0000","last_undegraded":"2024-04-25T18:43:14.908454+0000","last_fullsized":"2024-04-25T18:43:14.908454+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676447+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:42:54.676447+0000","last_peered":"2024-04-25T18:42:54.676447+0000","last_clean":"2024-04-25T18:42:54.676447+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:42:54.676447+0000","last_undegraded":"2024-04-25T18:42:54.676447+0000","last_fullsized":"2024-04-25T18:42:54.676447+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112382+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:42:55.112382+0000","last_peered":"2024-04-25T18:42:55.112382+0000","last_clean":"2024-04-25T18:42:55.112382+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:42:55.112382+0000","last_undegraded":"2024-04-25T18:42:55.112382+0000","last_fullsized":"2024-04-25T18:42:55.112382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685183+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:42:54.685183+0000","last_peered":"2024-04-25T18:42:54.685183+0000","last_clean":"2024-04-25T18:42:54.685183+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:42:54.685183+0000","last_undegraded":"2024-04-25T18:42:54.685183+0000","last_fullsized":"2024-04-25T18:42:54.685183+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676674+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:42:54.676674+0000","last_peered":"2024-04-25T18:42:54.676674+0000","last_clean":"2024-04-25T18:42:54.676674+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:42:54.676674+0000","last_undegraded":"2024-04-25T18:42:54.676674+0000","last_fullsized":"2024-04-25T18:42:54.676674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675599+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:42:54.675599+0000","last_peered":"2024-04-25T18:42:54.675599+0000","last_clean":"2024-04-25T18:42:54.675599+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:42:54.675599+0000","last_undegraded":"2024-04-25T18:42:54.675599+0000","last_fullsized":"2024-04-25T18:42:54.675599+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674978+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:42:54.674978+0000","last_peered":"2024-04-25T18:42:54.674978+0000","last_clean":"2024-04-25T18:42:54.674978+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:42:54.674978+0000","last_undegraded":"2024-04-25T18:42:54.674978+0000","last_fullsized":"2024-04-25T18:42:54.674978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676666+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:42:54.676666+0000","last_peered":"2024-04-25T18:42:54.676666+0000","last_clean":"2024-04-25T18:42:54.676666+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:42:54.676666+0000","last_undegraded":"2024-04-25T18:42:54.676666+0000","last_fullsized":"2024-04-25T18:42:54.676666+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673567+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:42:54.673567+0000","last_peered":"2024-04-25T18:42:54.673567+0000","last_clean":"2024-04-25T18:42:54.673567+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:42:54.673567+0000","last_undegraded":"2024-04-25T18:42:54.673567+0000","last_fullsized":"2024-04-25T18:42:54.673567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675574+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:42:54.675574+0000","last_peered":"2024-04-25T18:42:54.675574+0000","last_clean":"2024-04-25T18:42:54.675574+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:42:54.675574+0000","last_undegraded":"2024-04-25T18:42:54.675574+0000","last_fullsized":"2024-04-25T18:42:54.675574+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.390330+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:42:55.390330+0000","last_peered":"2024-04-25T18:42:55.390330+0000","last_clean":"2024-04-25T18:42:55.390330+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:42:55.390330+0000","last_undegraded":"2024-04-25T18:42:55.390330+0000","last_fullsized":"2024-04-25T18:42:55.390330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679258+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:42:54.679258+0000","last_peered":"2024-04-25T18:42:54.679258+0000","last_clean":"2024-04-25T18:42:54.679258+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:42:54.679258+0000","last_undegraded":"2024-04-25T18:42:54.679258+0000","last_fullsized":"2024-04-25T18:42:54.679258+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673792+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:42:54.673792+0000","last_peered":"2024-04-25T18:42:54.673792+0000","last_clean":"2024-04-25T18:42:54.673792+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:42:54.673792+0000","last_undegraded":"2024-04-25T18:42:54.673792+0000","last_fullsized":"2024-04-25T18:42:54.673792+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679648+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:42:54.679648+0000","last_peered":"2024-04-25T18:42:54.679648+0000","last_clean":"2024-04-25T18:42:54.679648+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:42:54.679648+0000","last_undegraded":"2024-04-25T18:42:54.679648+0000","last_fullsized":"2024-04-25T18:42:54.679648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678187+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:42:54.678187+0000","last_peered":"2024-04-25T18:42:54.678187+0000","last_clean":"2024-04-25T18:42:54.678187+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:42:54.678187+0000","last_undegraded":"2024-04-25T18:42:54.678187+0000","last_fullsized":"2024-04-25T18:42:54.678187+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679185+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:42:54.679185+0000","last_peered":"2024-04-25T18:42:54.679185+0000","last_clean":"2024-04-25T18:42:54.679185+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:42:54.679185+0000","last_undegraded":"2024-04-25T18:42:54.679185+0000","last_fullsized":"2024-04-25T18:42:54.679185+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382862+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:42:55.382862+0000","last_peered":"2024-04-25T18:42:55.382862+0000","last_clean":"2024-04-25T18:42:55.382862+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:42:55.382862+0000","last_undegraded":"2024-04-25T18:42:55.382862+0000","last_fullsized":"2024-04-25T18:42:55.382862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679248+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:42:54.679248+0000","last_peered":"2024-04-25T18:42:54.679248+0000","last_clean":"2024-04-25T18:42:54.679248+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:42:54.679248+0000","last_undegraded":"2024-04-25T18:42:54.679248+0000","last_fullsized":"2024-04-25T18:42:54.679248+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676785+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:42:54.676785+0000","last_peered":"2024-04-25T18:42:54.676785+0000","last_clean":"2024-04-25T18:42:54.676785+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:42:54.676785+0000","last_undegraded":"2024-04-25T18:42:54.676785+0000","last_fullsized":"2024-04-25T18:42:54.676785+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673579+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:42:54.673579+0000","last_peered":"2024-04-25T18:42:54.673579+0000","last_clean":"2024-04-25T18:42:54.673579+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:42:54.673579+0000","last_undegraded":"2024-04-25T18:42:54.673579+0000","last_fullsized":"2024-04-25T18:42:54.673579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673482+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:42:54.673482+0000","last_peered":"2024-04-25T18:42:54.673482+0000","last_clean":"2024-04-25T18:42:54.673482+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:42:54.673482+0000","last_undegraded":"2024-04-25T18:42:54.673482+0000","last_fullsized":"2024-04-25T18:42:54.673482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684425+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:42:54.684425+0000","last_peered":"2024-04-25T18:42:54.684425+0000","last_clean":"2024-04-25T18:42:54.684425+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:42:54.684425+0000","last_undegraded":"2024-04-25T18:42:54.684425+0000","last_fullsized":"2024-04-25T18:42:54.684425+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382690+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:42:55.382690+0000","last_peered":"2024-04-25T18:42:55.382690+0000","last_clean":"2024-04-25T18:42:55.382690+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:42:55.382690+0000","last_undegraded":"2024-04-25T18:42:55.382690+0000","last_fullsized":"2024-04-25T18:42:55.382690+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675600+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:42:54.675600+0000","last_peered":"2024-04-25T18:42:54.675600+0000","last_clean":"2024-04-25T18:42:54.675600+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:42:54.675600+0000","last_undegraded":"2024-04-25T18:42:54.675600+0000","last_fullsized":"2024-04-25T18:42:54.675600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674045+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:42:54.674045+0000","last_peered":"2024-04-25T18:42:54.674045+0000","last_clean":"2024-04-25T18:42:54.674045+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:42:54.674045+0000","last_undegraded":"2024-04-25T18:42:54.674045+0000","last_fullsized":"2024-04-25T18:42:54.674045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676436+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:42:54.676436+0000","last_peered":"2024-04-25T18:42:54.676436+0000","last_clean":"2024-04-25T18:42:54.676436+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:42:54.676436+0000","last_undegraded":"2024-04-25T18:42:54.676436+0000","last_fullsized":"2024-04-25T18:42:54.676436+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677187+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:42:54.677187+0000","last_peered":"2024-04-25T18:42:54.677187+0000","last_clean":"2024-04-25T18:42:54.677187+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:42:54.677187+0000","last_undegraded":"2024-04-25T18:42:54.677187+0000","last_fullsized":"2024-04-25T18:42:54.677187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674748+0000","last_change":"2024-04-25T18:42:28.902432+0000","last_active":"2024-04-25T18:42:54.674748+0000","last_peered":"2024-04-25T18:42:54.674748+0000","last_clean":"2024-04-25T18:42:54.674748+0000","last_became_active":"2024-04-25T18:42:28.902085+0000","last_became_peered":"2024-04-25T18:42:28.902085+0000","last_unstale":"2024-04-25T18:42:54.674748+0000","last_undegraded":"2024-04-25T18:42:54.674748+0000","last_fullsized":"2024-04-25T18:42:54.674748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:30.205627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676882+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:42:54.676882+0000","last_peered":"2024-04-25T18:42:54.676882+0000","last_clean":"2024-04-25T18:42:54.676882+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:42:54.676882+0000","last_undegraded":"2024-04-25T18:42:54.676882+0000","last_fullsized":"2024-04-25T18:42:54.676882+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382944+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:42:55.382944+0000","last_peered":"2024-04-25T18:42:55.382944+0000","last_clean":"2024-04-25T18:42:55.382944+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:42:55.382944+0000","last_undegraded":"2024-04-25T18:42:55.382944+0000","last_fullsized":"2024-04-25T18:42:55.382944+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.683581+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:42:54.683581+0000","last_peered":"2024-04-25T18:42:54.683581+0000","last_clean":"2024-04-25T18:42:54.683581+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:42:54.683581+0000","last_undegraded":"2024-04-25T18:42:54.683581+0000","last_fullsized":"2024-04-25T18:42:54.683581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59506,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":204,"num_read_kb":162,"num_write":161,"num_write_kb":105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":187032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252894,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":497006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514530,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41168,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703984,"statfs":{"total":95995035648,"available":95952879616,"internally_reserved":0,"allocated":1712128,"data_stored":1144271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678056,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":497130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939696,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":496356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234036,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1630208,"data_stored":1088980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397561,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":496519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":556656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":557312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117065,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":496830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855428,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":497459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659201,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32780,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712372,"statfs":{"total":95995035648,"available":95961468928,"internally_reserved":0,"allocated":1642496,"data_stored":1088273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":44,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:43:30.040 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:43:30.508 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:30 smithi063 ceph-mon[28006]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:30.508 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:30 smithi063 ceph-mon[28006]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T18:43:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:30 smithi114 ceph-mon[32988]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:30 smithi114 ceph-mon[32988]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T18:43:30.571 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:30 smithi070 ceph-mon[32820]: from='client.24664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:30 smithi070 ceph-mon[32820]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-04-25T18:43:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:32 smithi114 ceph-mon[32988]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1022 B/s wr, 5 op/s 2024-04-25T18:43:32.595 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:32 smithi063 ceph-mon[28006]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1022 B/s wr, 5 op/s 2024-04-25T18:43:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:32 smithi070 ceph-mon[32820]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1022 B/s wr, 5 op/s 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 -- 172.21.15.63:0/42762870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 msgr2=0x7efbfc1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 --2- 172.21.15.63:0/42762870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc1098c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efbe80099c0 tx=0x7efbe8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 -- 172.21.15.63:0/42762870 shutdown_connections 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 --2- 172.21.15.63:0/42762870 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc109fa0 0x7efbfc111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 --2- 172.21.15.63:0/42762870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc104ec0 0x7efbfc1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 --2- 172.21.15.63:0/42762870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc1098c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 -- 172.21.15.63:0/42762870 >> 172.21.15.63:0/42762870 conn(0x7efbfc100750 msgr2=0x7efbfc102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.769+0000 7efc03400700 1 -- 172.21.15.63:0/42762870 shutdown_connections 2024-04-25T18:43:32.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.770+0000 7efc03400700 1 -- 172.21.15.63:0/42762870 wait complete. 2024-04-25T18:43:32.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.771+0000 7efc03400700 1 Processor -- start 2024-04-25T18:43:32.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.771+0000 7efc03400700 1 -- start start 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 0x7efbfc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc114e80 con 0x7efbfc105870 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc114d00 con 0x7efbfc109fa0 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc03400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc115000 con 0x7efbfc104ec0 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc00a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:32.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efbfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:32.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc00a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34964/0 (socket says 172.21.15.63:34964) 2024-04-25T18:43:32.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc00a00700 1 -- 172.21.15.63:0/1306273721 learned_addr learned my addr 172.21.15.63:0/1306273721 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:32.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.772+0000 7efc01400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 0x7efbfc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 -- 172.21.15.63:0/1306273721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 msgr2=0x7efbfc19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 0x7efbfc19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 -- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 msgr2=0x7efbfc19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 -- 172.21.15.63:0/1306273721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc1a0350 con 0x7efbfc104ec0 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc01400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 0x7efbfc19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc00a00700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9a0 tx=0x7efbf000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf000ca10 con 0x7efbfc104ec0 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efbf000cbb0 con 0x7efbfc104ec0 2024-04-25T18:43:32.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbfc1a0640 con 0x7efbfc104ec0 2024-04-25T18:43:32.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.773+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0018680 con 0x7efbfc104ec0 2024-04-25T18:43:32.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.774+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbfc1a7ec0 con 0x7efbfc104ec0 2024-04-25T18:43:32.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.775+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbfc069730 con 0x7efbfc104ec0 2024-04-25T18:43:32.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.777+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efbf0010430 con 0x7efbfc104ec0 2024-04-25T18:43:32.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.779+0000 7efbf9600700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 0x7efbe40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:32.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.779+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7efbf00d0a10 con 0x7efbfc104ec0 2024-04-25T18:43:32.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.779+0000 7efbfbe00700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 0x7efbe40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:32.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.780+0000 7efbfbe00700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 0x7efbe40a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efbe8000c00 tx=0x7efbe8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:32.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:32.780+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efbf009d230 con 0x7efbfc104ec0 2024-04-25T18:43:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.018+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efbfc062e80 con 0x7efbe40a3d50 2024-04-25T18:43:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.042+0000 7efbf9600700 1 -- 172.21.15.63:0/1306273721 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258735 (secure 0 0 0) 0x7efbfc062e80 con 0x7efbe40a3d50 2024-04-25T18:43:33.048 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 msgr2=0x7efbe40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 0x7efbe40a6210 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efbe8000c00 tx=0x7efbe8005fb0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 msgr2=0x7efbfc19b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9a0 tx=0x7efbf000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 shutdown_connections 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7efbe40a3d50 0x7efbe40a6210 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efbfc104ec0 0x7efbfc19b390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efbfc109fa0 0x7efbfc19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 --2- 172.21.15.63:0/1306273721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efbfc105870 0x7efbfc19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:33.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.048+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 >> 172.21.15.63:0/1306273721 conn(0x7efbfc100750 msgr2=0x7efbfc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:33.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.049+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 shutdown_connections 2024-04-25T18:43:33.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:33.049+0000 7efc03400700 1 -- 172.21.15.63:0/1306273721 wait complete. 2024-04-25T18:43:33.051 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:43:33.970 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-04-25T18:43:31.151558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654365,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":370,"num_read_kb":301,"num_write":347,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388200,"kb_used_data":14076,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553624,"statfs":{"total":1151940427776,"available":1151542910976,"internally_reserved":0,"allocated":14413824,"data_stored":7918839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9067,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":40,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011775"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675644+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:42:54.675644+0000","last_peered":"2024-04-25T18:42:54.675644+0000","last_clean":"2024-04-25T18:42:54.675644+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:42:54.675644+0000","last_undegraded":"2024-04-25T18:42:54.675644+0000","last_fullsized":"2024-04-25T18:42:54.675644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685047+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:42:54.685047+0000","last_peered":"2024-04-25T18:42:54.685047+0000","last_clean":"2024-04-25T18:42:54.685047+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:42:54.685047+0000","last_undegraded":"2024-04-25T18:42:54.685047+0000","last_fullsized":"2024-04-25T18:42:54.685047+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684922+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:42:54.684922+0000","last_peered":"2024-04-25T18:42:54.684922+0000","last_clean":"2024-04-25T18:42:54.684922+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:42:54.684922+0000","last_undegraded":"2024-04-25T18:42:54.684922+0000","last_fullsized":"2024-04-25T18:42:54.684922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112496+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:42:55.112496+0000","last_peered":"2024-04-25T18:42:55.112496+0000","last_clean":"2024-04-25T18:42:55.112496+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:42:55.112496+0000","last_undegraded":"2024-04-25T18:42:55.112496+0000","last_fullsized":"2024-04-25T18:42:55.112496+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677379+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:42:54.677379+0000","last_peered":"2024-04-25T18:42:54.677379+0000","last_clean":"2024-04-25T18:42:54.677379+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:42:54.677379+0000","last_undegraded":"2024-04-25T18:42:54.677379+0000","last_fullsized":"2024-04-25T18:42:54.677379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673197+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:42:54.673197+0000","last_peered":"2024-04-25T18:42:54.673197+0000","last_clean":"2024-04-25T18:42:54.673197+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:42:54.673197+0000","last_undegraded":"2024-04-25T18:42:54.673197+0000","last_fullsized":"2024-04-25T18:42:54.673197+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678909+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:42:54.678909+0000","last_peered":"2024-04-25T18:42:54.678909+0000","last_clean":"2024-04-25T18:42:54.678909+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:42:54.678909+0000","last_undegraded":"2024-04-25T18:42:54.678909+0000","last_fullsized":"2024-04-25T18:42:54.678909+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674648+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:42:54.674648+0000","last_peered":"2024-04-25T18:42:54.674648+0000","last_clean":"2024-04-25T18:42:54.674648+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:42:54.674648+0000","last_undegraded":"2024-04-25T18:42:54.674648+0000","last_fullsized":"2024-04-25T18:42:54.674648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677434+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:42:54.677434+0000","last_peered":"2024-04-25T18:42:54.677434+0000","last_clean":"2024-04-25T18:42:54.677434+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:42:54.677434+0000","last_undegraded":"2024-04-25T18:42:54.677434+0000","last_fullsized":"2024-04-25T18:42:54.677434+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673511+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:42:54.673511+0000","last_peered":"2024-04-25T18:42:54.673511+0000","last_clean":"2024-04-25T18:42:54.673511+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:42:54.673511+0000","last_undegraded":"2024-04-25T18:42:54.673511+0000","last_fullsized":"2024-04-25T18:42:54.673511+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679526+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:42:54.679526+0000","last_peered":"2024-04-25T18:42:54.679526+0000","last_clean":"2024-04-25T18:42:54.679526+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:42:54.679526+0000","last_undegraded":"2024-04-25T18:42:54.679526+0000","last_fullsized":"2024-04-25T18:42:54.679526+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112449+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:42:55.112449+0000","last_peered":"2024-04-25T18:42:55.112449+0000","last_clean":"2024-04-25T18:42:55.112449+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:42:55.112449+0000","last_undegraded":"2024-04-25T18:42:55.112449+0000","last_fullsized":"2024-04-25T18:42:55.112449+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382799+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:42:55.382799+0000","last_peered":"2024-04-25T18:42:55.382799+0000","last_clean":"2024-04-25T18:42:55.382799+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:42:55.382799+0000","last_undegraded":"2024-04-25T18:42:55.382799+0000","last_fullsized":"2024-04-25T18:42:55.382799+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684494+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:42:54.684494+0000","last_peered":"2024-04-25T18:42:54.684494+0000","last_clean":"2024-04-25T18:42:54.684494+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:42:54.684494+0000","last_undegraded":"2024-04-25T18:42:54.684494+0000","last_fullsized":"2024-04-25T18:42:54.684494+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.683167+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:42:54.683167+0000","last_peered":"2024-04-25T18:42:54.683167+0000","last_clean":"2024-04-25T18:42:54.683167+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:42:54.683167+0000","last_undegraded":"2024-04-25T18:42:54.683167+0000","last_fullsized":"2024-04-25T18:42:54.683167+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676652+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:42:54.676652+0000","last_peered":"2024-04-25T18:42:54.676652+0000","last_clean":"2024-04-25T18:42:54.676652+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:42:54.676652+0000","last_undegraded":"2024-04-25T18:42:54.676652+0000","last_fullsized":"2024-04-25T18:42:54.676652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675880+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:42:54.675880+0000","last_peered":"2024-04-25T18:42:54.675880+0000","last_clean":"2024-04-25T18:42:54.675880+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:42:54.675880+0000","last_undegraded":"2024-04-25T18:42:54.675880+0000","last_fullsized":"2024-04-25T18:42:54.675880+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680693+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:42:54.680693+0000","last_peered":"2024-04-25T18:42:54.680693+0000","last_clean":"2024-04-25T18:42:54.680693+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:42:54.680693+0000","last_undegraded":"2024-04-25T18:42:54.680693+0000","last_fullsized":"2024-04-25T18:42:54.680693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673892+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:42:54.673892+0000","last_peered":"2024-04-25T18:42:54.673892+0000","last_clean":"2024-04-25T18:42:54.673892+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:42:54.673892+0000","last_undegraded":"2024-04-25T18:42:54.673892+0000","last_fullsized":"2024-04-25T18:42:54.673892+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.699790+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:42:54.699790+0000","last_peered":"2024-04-25T18:42:54.699790+0000","last_clean":"2024-04-25T18:42:54.699790+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:42:54.699790+0000","last_undegraded":"2024-04-25T18:42:54.699790+0000","last_fullsized":"2024-04-25T18:42:54.699790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676165+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:42:54.676165+0000","last_peered":"2024-04-25T18:42:54.676165+0000","last_clean":"2024-04-25T18:42:54.676165+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:42:54.676165+0000","last_undegraded":"2024-04-25T18:42:54.676165+0000","last_fullsized":"2024-04-25T18:42:54.676165+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676461+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:42:54.676461+0000","last_peered":"2024-04-25T18:42:54.676461+0000","last_clean":"2024-04-25T18:42:54.676461+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:42:54.676461+0000","last_undegraded":"2024-04-25T18:42:54.676461+0000","last_fullsized":"2024-04-25T18:42:54.676461+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685320+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:42:54.685320+0000","last_peered":"2024-04-25T18:42:54.685320+0000","last_clean":"2024-04-25T18:42:54.685320+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:42:54.685320+0000","last_undegraded":"2024-04-25T18:42:54.685320+0000","last_fullsized":"2024-04-25T18:42:54.685320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676702+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:42:54.676702+0000","last_peered":"2024-04-25T18:42:54.676702+0000","last_clean":"2024-04-25T18:42:54.676702+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:42:54.676702+0000","last_undegraded":"2024-04-25T18:42:54.676702+0000","last_fullsized":"2024-04-25T18:42:54.676702+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676905+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:42:54.676905+0000","last_peered":"2024-04-25T18:42:54.676905+0000","last_clean":"2024-04-25T18:42:54.676905+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:42:54.676905+0000","last_undegraded":"2024-04-25T18:42:54.676905+0000","last_fullsized":"2024-04-25T18:42:54.676905+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677652+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:42:54.677652+0000","last_peered":"2024-04-25T18:42:54.677652+0000","last_clean":"2024-04-25T18:42:54.677652+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:42:54.677652+0000","last_undegraded":"2024-04-25T18:42:54.677652+0000","last_fullsized":"2024-04-25T18:42:54.677652+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.681497+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:42:54.681497+0000","last_peered":"2024-04-25T18:42:54.681497+0000","last_clean":"2024-04-25T18:42:54.681497+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:42:54.681497+0000","last_undegraded":"2024-04-25T18:42:54.681497+0000","last_fullsized":"2024-04-25T18:42:54.681497+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:26.677156+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:43:26.677156+0000","last_peered":"2024-04-25T18:43:26.677156+0000","last_clean":"2024-04-25T18:43:26.677156+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:43:26.677156+0000","last_undegraded":"2024-04-25T18:43:26.677156+0000","last_fullsized":"2024-04-25T18:43:26.677156+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.384178+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:42:55.384178+0000","last_peered":"2024-04-25T18:42:55.384178+0000","last_clean":"2024-04-25T18:42:55.384178+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:42:55.384178+0000","last_undegraded":"2024-04-25T18:42:55.384178+0000","last_fullsized":"2024-04-25T18:42:55.384178+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673085+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:42:54.673085+0000","last_peered":"2024-04-25T18:42:54.673085+0000","last_clean":"2024-04-25T18:42:54.673085+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:42:54.673085+0000","last_undegraded":"2024-04-25T18:42:54.673085+0000","last_fullsized":"2024-04-25T18:42:54.673085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677767+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:42:54.677767+0000","last_peered":"2024-04-25T18:42:54.677767+0000","last_clean":"2024-04-25T18:42:54.677767+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:42:54.677767+0000","last_undegraded":"2024-04-25T18:42:54.677767+0000","last_fullsized":"2024-04-25T18:42:54.677767+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382290+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:42:55.382290+0000","last_peered":"2024-04-25T18:42:55.382290+0000","last_clean":"2024-04-25T18:42:55.382290+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:42:55.382290+0000","last_undegraded":"2024-04-25T18:42:55.382290+0000","last_fullsized":"2024-04-25T18:42:55.382290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674960+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:42:54.674960+0000","last_peered":"2024-04-25T18:42:54.674960+0000","last_clean":"2024-04-25T18:42:54.674960+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:42:54.674960+0000","last_undegraded":"2024-04-25T18:42:54.674960+0000","last_fullsized":"2024-04-25T18:42:54.674960+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677070+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:42:54.677070+0000","last_peered":"2024-04-25T18:42:54.677070+0000","last_clean":"2024-04-25T18:42:54.677070+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:42:54.677070+0000","last_undegraded":"2024-04-25T18:42:54.677070+0000","last_fullsized":"2024-04-25T18:42:54.677070+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680411+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:42:54.680411+0000","last_peered":"2024-04-25T18:42:54.680411+0000","last_clean":"2024-04-25T18:42:54.680411+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:42:54.680411+0000","last_undegraded":"2024-04-25T18:42:54.680411+0000","last_fullsized":"2024-04-25T18:42:54.680411+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:26.676396+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:43:26.676396+0000","last_peered":"2024-04-25T18:43:26.676396+0000","last_clean":"2024-04-25T18:43:26.676396+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:43:26.676396+0000","last_undegraded":"2024-04-25T18:43:26.676396+0000","last_fullsized":"2024-04-25T18:43:26.676396+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677259+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:42:54.677259+0000","last_peered":"2024-04-25T18:42:54.677259+0000","last_clean":"2024-04-25T18:42:54.677259+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:42:54.677259+0000","last_undegraded":"2024-04-25T18:42:54.677259+0000","last_fullsized":"2024-04-25T18:42:54.677259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":16,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.118076+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:42:55.118076+0000","last_peered":"2024-04-25T18:42:55.118076+0000","last_clean":"2024-04-25T18:42:55.118076+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:42:55.118076+0000","last_undegraded":"2024-04-25T18:42:55.118076+0000","last_fullsized":"2024-04-25T18:42:55.118076+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685396+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:42:54.685396+0000","last_peered":"2024-04-25T18:42:54.685396+0000","last_clean":"2024-04-25T18:42:54.685396+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:42:54.685396+0000","last_undegraded":"2024-04-25T18:42:54.685396+0000","last_fullsized":"2024-04-25T18:42:54.685396+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678632+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:42:54.678632+0000","last_peered":"2024-04-25T18:42:54.678632+0000","last_clean":"2024-04-25T18:42:54.678632+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:42:54.678632+0000","last_undegraded":"2024-04-25T18:42:54.678632+0000","last_fullsized":"2024-04-25T18:42:54.678632+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680465+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:42:54.680465+0000","last_peered":"2024-04-25T18:42:54.680465+0000","last_clean":"2024-04-25T18:42:54.680465+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:42:54.680465+0000","last_undegraded":"2024-04-25T18:42:54.680465+0000","last_fullsized":"2024-04-25T18:42:54.680465+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"94'29","reported_seq":134,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:27.678779+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:43:27.678779+0000","last_peered":"2024-04-25T18:43:27.678779+0000","last_clean":"2024-04-25T18:43:27.678779+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:43:27.678779+0000","last_undegraded":"2024-04-25T18:43:27.678779+0000","last_fullsized":"2024-04-25T18:43:27.678779+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59762,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":79,"num_read_kb":97,"num_write":30,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676420+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:42:54.676420+0000","last_peered":"2024-04-25T18:42:54.676420+0000","last_clean":"2024-04-25T18:42:54.676420+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:42:54.676420+0000","last_undegraded":"2024-04-25T18:42:54.676420+0000","last_fullsized":"2024-04-25T18:42:54.676420+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673719+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:42:54.673719+0000","last_peered":"2024-04-25T18:42:54.673719+0000","last_clean":"2024-04-25T18:42:54.673719+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:42:54.673719+0000","last_undegraded":"2024-04-25T18:42:54.673719+0000","last_fullsized":"2024-04-25T18:42:54.673719+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382354+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:42:55.382354+0000","last_peered":"2024-04-25T18:42:55.382354+0000","last_clean":"2024-04-25T18:42:55.382354+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:42:55.382354+0000","last_undegraded":"2024-04-25T18:42:55.382354+0000","last_fullsized":"2024-04-25T18:42:55.382354+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677488+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:42:54.677488+0000","last_peered":"2024-04-25T18:42:54.677488+0000","last_clean":"2024-04-25T18:42:54.677488+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:42:54.677488+0000","last_undegraded":"2024-04-25T18:42:54.677488+0000","last_fullsized":"2024-04-25T18:42:54.677488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679419+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:42:54.679419+0000","last_peered":"2024-04-25T18:42:54.679419+0000","last_clean":"2024-04-25T18:42:54.679419+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:42:54.679419+0000","last_undegraded":"2024-04-25T18:42:54.679419+0000","last_fullsized":"2024-04-25T18:42:54.679419+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"94'4","reported_seq":68,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:26.694020+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:43:26.694020+0000","last_peered":"2024-04-25T18:43:26.694020+0000","last_clean":"2024-04-25T18:43:26.694020+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:43:26.694020+0000","last_undegraded":"2024-04-25T18:43:26.694020+0000","last_fullsized":"2024-04-25T18:43:26.694020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678855+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:42:54.678855+0000","last_peered":"2024-04-25T18:42:54.678855+0000","last_clean":"2024-04-25T18:42:54.678855+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:42:54.678855+0000","last_undegraded":"2024-04-25T18:42:54.678855+0000","last_fullsized":"2024-04-25T18:42:54.678855+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.144779+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:42:55.144779+0000","last_peered":"2024-04-25T18:42:55.144779+0000","last_clean":"2024-04-25T18:42:55.144779+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:42:55.144779+0000","last_undegraded":"2024-04-25T18:42:55.144779+0000","last_fullsized":"2024-04-25T18:42:55.144779+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677070+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:42:54.677070+0000","last_peered":"2024-04-25T18:42:54.677070+0000","last_clean":"2024-04-25T18:42:54.677070+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:42:54.677070+0000","last_undegraded":"2024-04-25T18:42:54.677070+0000","last_fullsized":"2024-04-25T18:42:54.677070+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675999+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:42:54.675999+0000","last_peered":"2024-04-25T18:42:54.675999+0000","last_clean":"2024-04-25T18:42:54.675999+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:42:54.675999+0000","last_undegraded":"2024-04-25T18:42:54.675999+0000","last_fullsized":"2024-04-25T18:42:54.675999+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676579+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:42:54.676579+0000","last_peered":"2024-04-25T18:42:54.676579+0000","last_clean":"2024-04-25T18:42:54.676579+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:42:54.676579+0000","last_undegraded":"2024-04-25T18:42:54.676579+0000","last_fullsized":"2024-04-25T18:42:54.676579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.680841+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:42:54.680841+0000","last_peered":"2024-04-25T18:42:54.680841+0000","last_clean":"2024-04-25T18:42:54.680841+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:42:54.680841+0000","last_undegraded":"2024-04-25T18:42:54.680841+0000","last_fullsized":"2024-04-25T18:42:54.680841+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"94'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673405+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:42:54.673405+0000","last_peered":"2024-04-25T18:42:54.673405+0000","last_clean":"2024-04-25T18:42:54.673405+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:42:54.673405+0000","last_undegraded":"2024-04-25T18:42:54.673405+0000","last_fullsized":"2024-04-25T18:42:54.673405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679258+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:42:54.679258+0000","last_peered":"2024-04-25T18:42:54.679258+0000","last_clean":"2024-04-25T18:42:54.679258+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:42:54.679258+0000","last_undegraded":"2024-04-25T18:42:54.679258+0000","last_fullsized":"2024-04-25T18:42:54.679258+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673123+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:42:54.673123+0000","last_peered":"2024-04-25T18:42:54.673123+0000","last_clean":"2024-04-25T18:42:54.673123+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:42:54.673123+0000","last_undegraded":"2024-04-25T18:42:54.673123+0000","last_fullsized":"2024-04-25T18:42:54.673123+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677448+0000","last_change":"2024-04-25T18:42:28.876775+0000","last_active":"2024-04-25T18:42:54.677448+0000","last_peered":"2024-04-25T18:42:54.677448+0000","last_clean":"2024-04-25T18:42:54.677448+0000","last_became_active":"2024-04-25T18:42:28.876377+0000","last_became_peered":"2024-04-25T18:42:28.876377+0000","last_unstale":"2024-04-25T18:42:54.677448+0000","last_undegraded":"2024-04-25T18:42:54.677448+0000","last_fullsized":"2024-04-25T18:42:54.677448+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:23:34.927698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"92'4","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:26.693261+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:43:26.693261+0000","last_peered":"2024-04-25T18:43:26.693261+0000","last_clean":"2024-04-25T18:43:26.693261+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:43:26.693261+0000","last_undegraded":"2024-04-25T18:43:26.693261+0000","last_fullsized":"2024-04-25T18:43:26.693261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676068+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:42:54.676068+0000","last_peered":"2024-04-25T18:42:54.676068+0000","last_clean":"2024-04-25T18:42:54.676068+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:42:54.676068+0000","last_undegraded":"2024-04-25T18:42:54.676068+0000","last_fullsized":"2024-04-25T18:42:54.676068+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112350+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:42:55.112350+0000","last_peered":"2024-04-25T18:42:55.112350+0000","last_clean":"2024-04-25T18:42:55.112350+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:42:55.112350+0000","last_undegraded":"2024-04-25T18:42:55.112350+0000","last_fullsized":"2024-04-25T18:42:55.112350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":362,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679157+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:42:54.679157+0000","last_peered":"2024-04-25T18:42:54.679157+0000","last_clean":"2024-04-25T18:42:54.679157+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:42:54.679157+0000","last_undegraded":"2024-04-25T18:42:54.679157+0000","last_fullsized":"2024-04-25T18:42:54.679157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.381856+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:42:55.381856+0000","last_peered":"2024-04-25T18:42:55.381856+0000","last_clean":"2024-04-25T18:42:55.381856+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:42:55.381856+0000","last_undegraded":"2024-04-25T18:42:55.381856+0000","last_fullsized":"2024-04-25T18:42:55.381856+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679322+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:42:54.679322+0000","last_peered":"2024-04-25T18:42:54.679322+0000","last_clean":"2024-04-25T18:42:54.679322+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:42:54.679322+0000","last_undegraded":"2024-04-25T18:42:54.679322+0000","last_fullsized":"2024-04-25T18:42:54.679322+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676523+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:42:54.676523+0000","last_peered":"2024-04-25T18:42:54.676523+0000","last_clean":"2024-04-25T18:42:54.676523+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:42:54.676523+0000","last_undegraded":"2024-04-25T18:42:54.676523+0000","last_fullsized":"2024-04-25T18:42:54.676523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382850+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:42:55.382850+0000","last_peered":"2024-04-25T18:42:55.382850+0000","last_clean":"2024-04-25T18:42:55.382850+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:42:55.382850+0000","last_undegraded":"2024-04-25T18:42:55.382850+0000","last_fullsized":"2024-04-25T18:42:55.382850+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"94'11","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:43:18.907522+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:43:18.907522+0000","last_peered":"2024-04-25T18:43:18.907522+0000","last_clean":"2024-04-25T18:43:18.907522+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:43:18.907522+0000","last_undegraded":"2024-04-25T18:43:18.907522+0000","last_fullsized":"2024-04-25T18:43:18.907522+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676447+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:42:54.676447+0000","last_peered":"2024-04-25T18:42:54.676447+0000","last_clean":"2024-04-25T18:42:54.676447+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:42:54.676447+0000","last_undegraded":"2024-04-25T18:42:54.676447+0000","last_fullsized":"2024-04-25T18:42:54.676447+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.112382+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:42:55.112382+0000","last_peered":"2024-04-25T18:42:55.112382+0000","last_clean":"2024-04-25T18:42:55.112382+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:42:55.112382+0000","last_undegraded":"2024-04-25T18:42:55.112382+0000","last_fullsized":"2024-04-25T18:42:55.112382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.685183+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:42:54.685183+0000","last_peered":"2024-04-25T18:42:54.685183+0000","last_clean":"2024-04-25T18:42:54.685183+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:42:54.685183+0000","last_undegraded":"2024-04-25T18:42:54.685183+0000","last_fullsized":"2024-04-25T18:42:54.685183+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676674+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:42:54.676674+0000","last_peered":"2024-04-25T18:42:54.676674+0000","last_clean":"2024-04-25T18:42:54.676674+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:42:54.676674+0000","last_undegraded":"2024-04-25T18:42:54.676674+0000","last_fullsized":"2024-04-25T18:42:54.676674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675599+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:42:54.675599+0000","last_peered":"2024-04-25T18:42:54.675599+0000","last_clean":"2024-04-25T18:42:54.675599+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:42:54.675599+0000","last_undegraded":"2024-04-25T18:42:54.675599+0000","last_fullsized":"2024-04-25T18:42:54.675599+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674978+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:42:54.674978+0000","last_peered":"2024-04-25T18:42:54.674978+0000","last_clean":"2024-04-25T18:42:54.674978+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:42:54.674978+0000","last_undegraded":"2024-04-25T18:42:54.674978+0000","last_fullsized":"2024-04-25T18:42:54.674978+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676666+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:42:54.676666+0000","last_peered":"2024-04-25T18:42:54.676666+0000","last_clean":"2024-04-25T18:42:54.676666+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:42:54.676666+0000","last_undegraded":"2024-04-25T18:42:54.676666+0000","last_fullsized":"2024-04-25T18:42:54.676666+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673567+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:42:54.673567+0000","last_peered":"2024-04-25T18:42:54.673567+0000","last_clean":"2024-04-25T18:42:54.673567+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:42:54.673567+0000","last_undegraded":"2024-04-25T18:42:54.673567+0000","last_fullsized":"2024-04-25T18:42:54.673567+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675574+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:42:54.675574+0000","last_peered":"2024-04-25T18:42:54.675574+0000","last_clean":"2024-04-25T18:42:54.675574+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:42:54.675574+0000","last_undegraded":"2024-04-25T18:42:54.675574+0000","last_fullsized":"2024-04-25T18:42:54.675574+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.390330+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:42:55.390330+0000","last_peered":"2024-04-25T18:42:55.390330+0000","last_clean":"2024-04-25T18:42:55.390330+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:42:55.390330+0000","last_undegraded":"2024-04-25T18:42:55.390330+0000","last_fullsized":"2024-04-25T18:42:55.390330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679258+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:42:54.679258+0000","last_peered":"2024-04-25T18:42:54.679258+0000","last_clean":"2024-04-25T18:42:54.679258+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:42:54.679258+0000","last_undegraded":"2024-04-25T18:42:54.679258+0000","last_fullsized":"2024-04-25T18:42:54.679258+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673792+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:42:54.673792+0000","last_peered":"2024-04-25T18:42:54.673792+0000","last_clean":"2024-04-25T18:42:54.673792+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:42:54.673792+0000","last_undegraded":"2024-04-25T18:42:54.673792+0000","last_fullsized":"2024-04-25T18:42:54.673792+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679648+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:42:54.679648+0000","last_peered":"2024-04-25T18:42:54.679648+0000","last_clean":"2024-04-25T18:42:54.679648+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:42:54.679648+0000","last_undegraded":"2024-04-25T18:42:54.679648+0000","last_fullsized":"2024-04-25T18:42:54.679648+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.678187+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:42:54.678187+0000","last_peered":"2024-04-25T18:42:54.678187+0000","last_clean":"2024-04-25T18:42:54.678187+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:42:54.678187+0000","last_undegraded":"2024-04-25T18:42:54.678187+0000","last_fullsized":"2024-04-25T18:42:54.678187+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"94'1","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679185+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:42:54.679185+0000","last_peered":"2024-04-25T18:42:54.679185+0000","last_clean":"2024-04-25T18:42:54.679185+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:42:54.679185+0000","last_undegraded":"2024-04-25T18:42:54.679185+0000","last_fullsized":"2024-04-25T18:42:54.679185+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382862+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:42:55.382862+0000","last_peered":"2024-04-25T18:42:55.382862+0000","last_clean":"2024-04-25T18:42:55.382862+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:42:55.382862+0000","last_undegraded":"2024-04-25T18:42:55.382862+0000","last_fullsized":"2024-04-25T18:42:55.382862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.679248+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:42:54.679248+0000","last_peered":"2024-04-25T18:42:54.679248+0000","last_clean":"2024-04-25T18:42:54.679248+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:42:54.679248+0000","last_undegraded":"2024-04-25T18:42:54.679248+0000","last_fullsized":"2024-04-25T18:42:54.679248+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676785+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:42:54.676785+0000","last_peered":"2024-04-25T18:42:54.676785+0000","last_clean":"2024-04-25T18:42:54.676785+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:42:54.676785+0000","last_undegraded":"2024-04-25T18:42:54.676785+0000","last_fullsized":"2024-04-25T18:42:54.676785+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673579+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:42:54.673579+0000","last_peered":"2024-04-25T18:42:54.673579+0000","last_clean":"2024-04-25T18:42:54.673579+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:42:54.673579+0000","last_undegraded":"2024-04-25T18:42:54.673579+0000","last_fullsized":"2024-04-25T18:42:54.673579+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.673482+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:42:54.673482+0000","last_peered":"2024-04-25T18:42:54.673482+0000","last_clean":"2024-04-25T18:42:54.673482+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:42:54.673482+0000","last_undegraded":"2024-04-25T18:42:54.673482+0000","last_fullsized":"2024-04-25T18:42:54.673482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.684425+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:42:54.684425+0000","last_peered":"2024-04-25T18:42:54.684425+0000","last_clean":"2024-04-25T18:42:54.684425+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:42:54.684425+0000","last_undegraded":"2024-04-25T18:42:54.684425+0000","last_fullsized":"2024-04-25T18:42:54.684425+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382690+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:42:55.382690+0000","last_peered":"2024-04-25T18:42:55.382690+0000","last_clean":"2024-04-25T18:42:55.382690+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:42:55.382690+0000","last_undegraded":"2024-04-25T18:42:55.382690+0000","last_fullsized":"2024-04-25T18:42:55.382690+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.675600+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:42:54.675600+0000","last_peered":"2024-04-25T18:42:54.675600+0000","last_clean":"2024-04-25T18:42:54.675600+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:42:54.675600+0000","last_undegraded":"2024-04-25T18:42:54.675600+0000","last_fullsized":"2024-04-25T18:42:54.675600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674045+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:42:54.674045+0000","last_peered":"2024-04-25T18:42:54.674045+0000","last_clean":"2024-04-25T18:42:54.674045+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:42:54.674045+0000","last_undegraded":"2024-04-25T18:42:54.674045+0000","last_fullsized":"2024-04-25T18:42:54.674045+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676436+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:42:54.676436+0000","last_peered":"2024-04-25T18:42:54.676436+0000","last_clean":"2024-04-25T18:42:54.676436+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:42:54.676436+0000","last_undegraded":"2024-04-25T18:42:54.676436+0000","last_fullsized":"2024-04-25T18:42:54.676436+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.677187+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:42:54.677187+0000","last_peered":"2024-04-25T18:42:54.677187+0000","last_clean":"2024-04-25T18:42:54.677187+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:42:54.677187+0000","last_undegraded":"2024-04-25T18:42:54.677187+0000","last_fullsized":"2024-04-25T18:42:54.677187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.674748+0000","last_change":"2024-04-25T18:42:28.902432+0000","last_active":"2024-04-25T18:42:54.674748+0000","last_peered":"2024-04-25T18:42:54.674748+0000","last_clean":"2024-04-25T18:42:54.674748+0000","last_became_active":"2024-04-25T18:42:28.902085+0000","last_became_peered":"2024-04-25T18:42:28.902085+0000","last_unstale":"2024-04-25T18:42:54.674748+0000","last_undegraded":"2024-04-25T18:42:54.674748+0000","last_fullsized":"2024-04-25T18:42:54.674748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:30.205627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.676882+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:42:54.676882+0000","last_peered":"2024-04-25T18:42:54.676882+0000","last_clean":"2024-04-25T18:42:54.676882+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:42:54.676882+0000","last_undegraded":"2024-04-25T18:42:54.676882+0000","last_fullsized":"2024-04-25T18:42:54.676882+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:55.382944+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:42:55.382944+0000","last_peered":"2024-04-25T18:42:55.382944+0000","last_clean":"2024-04-25T18:42:55.382944+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:42:55.382944+0000","last_undegraded":"2024-04-25T18:42:55.382944+0000","last_fullsized":"2024-04-25T18:42:55.382944+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-25T18:42:54.683581+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:42:54.683581+0000","last_peered":"2024-04-25T18:42:54.683581+0000","last_clean":"2024-04-25T18:42:54.683581+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:42:54.683581+0000","last_undegraded":"2024-04-25T18:42:54.683581+0000","last_fullsized":"2024-04-25T18:42:54.683581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63763,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":234,"num_read_kb":182,"num_write":164,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252895,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":497006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514531,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41172,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703980,"statfs":{"total":95995035648,"available":95952875520,"internally_reserved":0,"allocated":1716224,"data_stored":1148528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678057,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":497130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939697,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":496356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234037,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1630208,"data_stored":1088980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397562,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":496519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":556656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":497790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411411,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":557312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117066,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":496830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":497459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32780,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712372,"statfs":{"total":95995035648,"available":95961468928,"internally_reserved":0,"allocated":1642496,"data_stored":1088273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":44,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:43:33.975 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T18:43:33.976 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T18:43:33.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T18:43:34.486 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:34 smithi114 ceph-mon[32988]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:34 smithi114 ceph-mon[32988]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-25T18:43:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:34 smithi063 ceph-mon[28006]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:34 smithi063 ceph-mon[28006]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-25T18:43:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:34 smithi070 ceph-mon[32820]: from='client.24670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:43:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:34 smithi070 ceph-mon[32820]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 7 op/s 2024-04-25T18:43:36.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:36 smithi063 ceph-mon[28006]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:36 smithi114 ceph-mon[32988]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:36 smithi070 ceph-mon[32820]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:36.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 -- 172.21.15.63:0/1435323032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 msgr2=0x7f2d7c1082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 --2- 172.21.15.63:0/1435323032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c1082d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009990 tx=0x7f2d68009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 -- 172.21.15.63:0/1435323032 shutdown_connections 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 --2- 172.21.15.63:0/1435323032 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 --2- 172.21.15.63:0/1435323032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c108810 0x7f2d7c10ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 --2- 172.21.15.63:0/1435323032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c1082d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 -- 172.21.15.63:0/1435323032 >> 172.21.15.63:0/1435323032 conn(0x7f2d7c0fcf60 msgr2=0x7f2d7c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 -- 172.21.15.63:0/1435323032 shutdown_connections 2024-04-25T18:43:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.804+0000 7f2d81400700 1 -- 172.21.15.63:0/1435323032 wait complete. 2024-04-25T18:43:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.805+0000 7f2d81400700 1 Processor -- start 2024-04-25T18:43:36.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 -- start start 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 0x7f2d7c101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10e570 con 0x7f2d7c105ee0 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10e3f0 con 0x7f2d7c1035b0 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d81400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7c10e6f0 con 0x7f2d7c108810 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d7a800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.806+0000 7f2d79e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d79e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41266/0 (socket says 172.21.15.63:41266) 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36132/0 (socket says 172.21.15.63:36132) 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d79e00700 1 -- 172.21.15.63:0/1678249934 learned_addr learned my addr 172.21.15.63:0/1678249934 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:36.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7b200700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 0x7f2d7c101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:36.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 -- 172.21.15.63:0/1678249934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 msgr2=0x7f2d7c101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:36.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 0x7f2d7c101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 -- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 msgr2=0x7f2d7c101560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:36.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7a800700 1 -- 172.21.15.63:0/1678249934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7c102350 con 0x7f2d7c1035b0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d7b200700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 0x7f2d7c101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.807+0000 7f2d79e00700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d7a800700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00e9f0 tx=0x7f2d6c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c004220 con 0x7f2d7c1035b0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d6c0043c0 con 0x7f2d7c1035b0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c005020 con 0x7f2d7c1035b0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7c1a2cb0 con 0x7f2d7c1035b0 2024-04-25T18:43:36.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.808+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d7c1a3170 con 0x7f2d7c1035b0 2024-04-25T18:43:36.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.809+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d7c050f80 con 0x7f2d7c1035b0 2024-04-25T18:43:36.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.812+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2d6c004750 con 0x7f2d7c1035b0 2024-04-25T18:43:36.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.813+0000 7f2d73400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 0x7f2d600a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:36.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.814+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f2d6c0d0640 con 0x7f2d7c1035b0 2024-04-25T18:43:36.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.814+0000 7f2d79e00700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 0x7f2d600a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:36.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.814+0000 7f2d79e00700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 0x7f2d600a62c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009750 tx=0x7f2d68005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:36.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:36.816+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d6c09ce60 con 0x7f2d7c1035b0 2024-04-25T18:43:37.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.145+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f2d7c02e050 con 0x7f2d7c1035b0 2024-04-25T18:43:37.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.154+0000 7f2d73400700 1 -- 172.21.15.63:0/1678249934 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f2d6c09c760 con 0x7f2d7c1035b0 2024-04-25T18:43:37.155 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-04-25T18:43:37.155 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBZpCpmuXvKCBAATBH2duJ6pW+NHOzzXEeAGg== 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.157+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 msgr2=0x7f2d600a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.157+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 0x7f2d600a62c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009750 tx=0x7f2d68005960 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 msgr2=0x7f2d7c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00e9f0 tx=0x7f2d6c00ed00 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 shutdown_connections 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2d600a3e00 0x7f2d600a62c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2d7c108810 0x7f2d7c101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2d7c1035b0 0x7f2d7c102f10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 --2- 172.21.15.63:0/1678249934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d7c105ee0 0x7f2d7c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 >> 172.21.15.63:0/1678249934 conn(0x7f2d7c0fcf60 msgr2=0x7f2d7c103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:37.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 shutdown_connections 2024-04-25T18:43:37.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:37.158+0000 7f2d81400700 1 -- 172.21.15.63:0/1678249934 wait complete. 2024-04-25T18:43:37.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:37 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1678249934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:37 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.544 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:37 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T18:43:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:37 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1678249934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:37 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:37 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T18:43:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:37 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1678249934' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:37 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T18:43:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:37 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T18:43:37.995 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:43:37.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T18:43:37.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T18:43:38.139 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T18:43:38.140 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T18:43:38.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T18:43:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:38 smithi114 ceph-mon[32988]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:38 smithi063 ceph-mon[28006]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:38.654 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:38 smithi070 ceph-mon[32820]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:43:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:40 smithi063 ceph-mon[28006]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:43:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:40 smithi114 ceph-mon[32988]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:43:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:40 smithi070 ceph-mon[32820]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:43:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 -- 172.21.15.63:0/576699636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d410e3b0 msgr2=0x7ff1d4115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/576699636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d410e3b0 0x7ff1d4115f80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009f90 tx=0x7ff1cc00b330 comp rx=0 tx=0).stop 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 -- 172.21.15.63:0/576699636 shutdown_connections 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/576699636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d410e3b0 0x7ff1d4115f80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/576699636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d41092d0 0x7ff1d41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/576699636 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d4109c80 0x7ff1d410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 -- 172.21.15.63:0/576699636 >> 172.21.15.63:0/576699636 conn(0x7ff1d4076210 msgr2=0x7ff1d4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 -- 172.21.15.63:0/576699636 shutdown_connections 2024-04-25T18:43:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.880+0000 7ff1dbc00700 1 -- 172.21.15.63:0/576699636 wait complete. 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.881+0000 7ff1dbc00700 1 Processor -- start 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 -- start start 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d4109c80 0x7ff1d419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 0x7ff1d41a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41192a0 con 0x7ff1d4109c80 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4119120 con 0x7ff1d41092d0 2024-04-25T18:43:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1dbc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4119420 con 0x7ff1d410e3b0 2024-04-25T18:43:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1d9200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1d9200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40796/0 (socket says 172.21.15.63:40796) 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1d9c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 0x7ff1d41a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1d9200700 1 -- 172.21.15.63:0/1580347073 learned_addr learned my addr 172.21.15.63:0/1580347073 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.882+0000 7ff1d3e00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d4109c80 0x7ff1d419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 -- 172.21.15.63:0/1580347073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 msgr2=0x7ff1d419f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 -- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d4109c80 msgr2=0x7ff1d419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d4109c80 0x7ff1d419fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 -- 172.21.15.63:0/1580347073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d41a4700 con 0x7ff1d410e3b0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9200700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d9c00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 0x7ff1d41a3fc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00b6c0 tx=0x7ff1cc00cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.883+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc021070 con 0x7ff1d410e3b0 2024-04-25T18:43:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.884+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1cc007d60 con 0x7ff1d410e3b0 2024-04-25T18:43:40.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.884+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d41a4990 con 0x7ff1d410e3b0 2024-04-25T18:43:40.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.884+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc005040 con 0x7ff1d410e3b0 2024-04-25T18:43:40.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.884+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d41ac270 con 0x7ff1d410e3b0 2024-04-25T18:43:40.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.885+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d4069730 con 0x7ff1d410e3b0 2024-04-25T18:43:40.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.887+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff1cc024070 con 0x7ff1d410e3b0 2024-04-25T18:43:40.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.888+0000 7ff1d1600700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 0x7ff1bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:40.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.888+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7ff1cc00f020 con 0x7ff1d410e3b0 2024-04-25T18:43:40.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.889+0000 7ff1d9200700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 0x7ff1bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:40.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.889+0000 7ff1d9200700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 0x7ff1bc0a6210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8011630 tx=0x7ff1c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:40.891+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1cc02b020 con 0x7ff1d410e3b0 2024-04-25T18:43:41.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.192+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff1d402df80 con 0x7ff1d410e3b0 2024-04-25T18:43:41.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.198+0000 7ff1d1600700 1 -- 172.21.15.63:0/1580347073 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264998 (secure 0 0 0) 0x7ff1cc09e720 con 0x7ff1d410e3b0 2024-04-25T18:43:41.200 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 msgr2=0x7ff1bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 0x7ff1bc0a6210 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8011630 tx=0x7ff1c8009260 comp rx=0 tx=0).stop 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 msgr2=0x7ff1d41a3fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 0x7ff1d41a3fc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00b6c0 tx=0x7ff1cc00cc30 comp rx=0 tx=0).stop 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 shutdown_connections 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ff1bc0a3d50 0x7ff1bc0a6210 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff1d410e3b0 0x7ff1d41a3fc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1d41092d0 0x7ff1d419f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:41.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 --2- 172.21.15.63:0/1580347073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1d4109c80 0x7ff1d419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:41.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 >> 172.21.15.63:0/1580347073 conn(0x7ff1d4076210 msgr2=0x7ff1d4076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:41.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 shutdown_connections 2024-04-25T18:43:41.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:41.205+0000 7ff1dbc00700 1 -- 172.21.15.63:0/1580347073 wait complete. 2024-04-25T18:43:41.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:41 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1580347073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T18:43:41.608 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:41 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1580347073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T18:43:41.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:41 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1580347073' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T18:43:42.053 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":3659595029},{"type":"v1","addr":"172.21.15.63:6801","nonce":3659595029}]},"active_addr":"172.21.15.63:6801/3659595029","active_change":"2024-04-25T18:33:52.960624+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.63:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3560456479}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2327478809}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3611634789}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3012688827}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2438238181}]}]} 2024-04-25T18:43:42.059 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T18:43:42.060 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T18:43:42.060 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:43:42.550 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:42 smithi114 ceph-mon[32988]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:43:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:42 smithi063 ceph-mon[28006]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:43:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:42 smithi070 ceph-mon[32820]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-25T18:43:44.555 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:44 smithi063 ceph-mon[28006]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:44.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:44 smithi114 ceph-mon[32988]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:44 smithi070 ceph-mon[32820]: pgmap v379: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 -- 172.21.15.63:0/2405606743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a406b0f0 msgr2=0x7f27a410f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 --2- 172.21.15.63:0/2405606743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a406b0f0 0x7f27a410f900 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f279c00b0a0 tx=0x7f279c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 -- 172.21.15.63:0/2405606743 shutdown_connections 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 --2- 172.21.15.63:0/2405606743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a4108a30 0x7f27a4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 --2- 172.21.15.63:0/2405606743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a406b0f0 0x7f27a410f900 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 --2- 172.21.15.63:0/2405606743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406a750 0x7f27a406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.847+0000 7f27a9200700 1 -- 172.21.15.63:0/2405606743 >> 172.21.15.63:0/2405606743 conn(0x7f27a40788f0 msgr2=0x7f27a4078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:44.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.848+0000 7f27a9200700 1 -- 172.21.15.63:0/2405606743 shutdown_connections 2024-04-25T18:43:44.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.848+0000 7f27a9200700 1 -- 172.21.15.63:0/2405606743 wait complete. 2024-04-25T18:43:44.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.849+0000 7f27a9200700 1 Processor -- start 2024-04-25T18:43:44.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.849+0000 7f27a9200700 1 -- start start 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 0x7f27a419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 0x7f27a419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 0x7f27a419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f27a4112c20 con 0x7f27a4108a30 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27a4112aa0 con 0x7f27a406b0f0 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a9200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27a4112da0 con 0x7f27a406a750 2024-04-25T18:43:44.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a1e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 0x7f27a419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a1e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 0x7f27a419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58722/0 (socket says 172.21.15.63:58722) 2024-04-25T18:43:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a1400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 0x7f27a419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.850+0000 7f27a1e00700 1 -- 172.21.15.63:0/3308381065 learned_addr learned my addr 172.21.15.63:0/3308381065 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a2800700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 0x7f27a419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:44.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 -- 172.21.15.63:0/3308381065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 msgr2=0x7f27a419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 0x7f27a419b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 -- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 msgr2=0x7f27a419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 0x7f27a419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 -- 172.21.15.63:0/3308381065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a41a0310 con 0x7f27a406b0f0 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a2800700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 0x7f27a419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.851+0000 7f27a1400700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 0x7f27a419b890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f279000e9f0 tx=0x7f279000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:44.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.852+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f279000cbd0 con 0x7f27a406b0f0 2024-04-25T18:43:44.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.852+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f279000cd70 con 0x7f27a406b0f0 2024-04-25T18:43:44.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.852+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790018680 con 0x7f27a406b0f0 2024-04-25T18:43:44.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.852+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a41a0600 con 0x7f27a406b0f0 2024-04-25T18:43:44.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.852+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a41a7e80 con 0x7f27a406b0f0 2024-04-25T18:43:44.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.853+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a4069730 con 0x7f27a406b0f0 2024-04-25T18:43:44.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.855+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2790010430 con 0x7f27a406b0f0 2024-04-25T18:43:44.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.857+0000 7f279aa00700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 0x7f27880a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:44.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.857+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7f27900d0b10 con 0x7f27a406b0f0 2024-04-25T18:43:44.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.857+0000 7f27a1e00700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 0x7f27880a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:44.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.858+0000 7f27a1e00700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 0x7f27880a62e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2794009ca0 tx=0x7f2794009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:44.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:44.859+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f279009d3b0 con 0x7f27a406b0f0 2024-04-25T18:43:45.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.089+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27a402df80 con 0x7f27a406b0f0 2024-04-25T18:43:45.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.091+0000 7f279aa00700 1 -- 172.21.15.63:0/3308381065 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22643 (secure 0 0 0) 0x7f279009ccb0 con 0x7f27a406b0f0 2024-04-25T18:43:45.093 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:45.093 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":94,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:42:54.660943+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.094+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 msgr2=0x7f27880a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.094+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 0x7f27880a62e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2794009ca0 tx=0x7f2794009260 comp rx=0 tx=0).stop 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.094+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 msgr2=0x7f27a419b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.094+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 0x7f27a419b890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f279000e9f0 tx=0x7f279000ed00 comp rx=0 tx=0).stop 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 shutdown_connections 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27880a3e20 0x7f27880a62e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27a406a750 0x7f27a419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27a4108a30 0x7f27a419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:45.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 --2- 172.21.15.63:0/3308381065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27a406b0f0 0x7f27a419b890 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:45.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 >> 172.21.15.63:0/3308381065 conn(0x7f27a40788f0 msgr2=0x7f27a410df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:45.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 shutdown_connections 2024-04-25T18:43:45.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:45.095+0000 7f27a9200700 1 -- 172.21.15.63:0/3308381065 wait complete. 2024-04-25T18:43:45.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:45 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3308381065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:45.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:45 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3308381065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:45.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:45 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3308381065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:46.066 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T18:43:46.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:43:46.533 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:46.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:46 smithi114 ceph-mon[32988]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:43:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:46 smithi063 ceph-mon[28006]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:43:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:46 smithi070 ceph-mon[32820]: pgmap v380: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:43:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:48 smithi114 ceph-mon[32988]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:48.586 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:48 smithi063 ceph-mon[28006]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:48 smithi070 ceph-mon[32820]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:48.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.898+0000 7fa431e00700 1 -- 172.21.15.63:0/1357333454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c102a90 msgr2=0x7fa42c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:48.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.898+0000 7fa431e00700 1 --2- 172.21.15.63:0/1357333454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c102a90 0x7fa42c102f10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa418009990 tx=0x7fa418009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 -- 172.21.15.63:0/1357333454 shutdown_connections 2024-04-25T18:43:48.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 --2- 172.21.15.63:0/1357333454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c103450 0x7fa42c10f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 --2- 172.21.15.63:0/1357333454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c108a50 0x7fa42c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 --2- 172.21.15.63:0/1357333454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c102a90 0x7fa42c102f10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 -- 172.21.15.63:0/1357333454 >> 172.21.15.63:0/1357333454 conn(0x7fa42c079ba0 msgr2=0x7fa42c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:48.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 -- 172.21.15.63:0/1357333454 shutdown_connections 2024-04-25T18:43:48.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.899+0000 7fa431e00700 1 -- 172.21.15.63:0/1357333454 wait complete. 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.900+0000 7fa431e00700 1 Processor -- start 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.900+0000 7fa431e00700 1 -- start start 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 0x7fa42c10f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c108a50 0x7fa42c1a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c112be0 con 0x7fa42c103450 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c112a60 con 0x7fa42c108a50 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa431e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c112d60 con 0x7fa42c102a90 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa42aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:48.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa42aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50866/0 (socket says 172.21.15.63:50866) 2024-04-25T18:43:48.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa42b400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c108a50 0x7fa42c1a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:48.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa42aa00700 1 -- 172.21.15.63:0/552532085 learned_addr learned my addr 172.21.15.63:0/552532085 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:48.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.901+0000 7fa42a000700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 0x7fa42c10f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 -- 172.21.15.63:0/552532085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c108a50 msgr2=0x7fa42c1a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c108a50 0x7fa42c1a4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 -- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 msgr2=0x7fa42c10f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 0x7fa42c10f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 -- 172.21.15.63:0/552532085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c1a4770 con 0x7fa42c102a90 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42a000700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 0x7fa42c10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa42aa00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9a0 tx=0x7fa41c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.902+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00cbd0 con 0x7fa42c102a90 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.903+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1a4a60 con 0x7fa42c102a90 2024-04-25T18:43:48.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.903+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa41c00cd70 con 0x7fa42c102a90 2024-04-25T18:43:48.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.903+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c018680 con 0x7fa42c102a90 2024-04-25T18:43:48.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.903+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1ac2e0 con 0x7fa42c102a90 2024-04-25T18:43:48.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.904+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c069730 con 0x7fa42c102a90 2024-04-25T18:43:48.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.906+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa41c010430 con 0x7fa42c102a90 2024-04-25T18:43:48.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.908+0000 7fa430a00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 0x7fa4100a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:48.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.908+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8227+0+0 (secure 0 0 0) 0x7fa41c0d0720 con 0x7fa42c102a90 2024-04-25T18:43:48.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.909+0000 7fa42a000700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 0x7fa4100a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.909+0000 7fa42a000700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 0x7fa4100a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b2e0 tx=0x7fa41801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:48.912+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa41c09cf40 con 0x7fa42c102a90 2024-04-25T18:43:49.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.143+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa42c02df80 con 0x7fa42c102a90 2024-04-25T18:43:49.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.145+0000 7fa430a00700 1 -- 172.21.15.63:0/552532085 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+22643 (secure 0 0 0) 0x7fa41c09c840 con 0x7fa42c102a90 2024-04-25T18:43:49.146 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:43:49.147 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":94,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:42:54.660943+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 msgr2=0x7fa4100a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 0x7fa4100a6210 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b2e0 tx=0x7fa41801a040 comp rx=0 tx=0).stop 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 msgr2=0x7fa42c10ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9a0 tx=0x7fa41c00ed60 comp rx=0 tx=0).stop 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 shutdown_connections 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa4100a3d50 0x7fa4100a6210 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa42c102a90 0x7fa42c10ec60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:49.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa42c108a50 0x7fa42c1a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:49.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 --2- 172.21.15.63:0/552532085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa42c103450 0x7fa42c10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:49.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 >> 172.21.15.63:0/552532085 conn(0x7fa42c079ba0 msgr2=0x7fa42c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:49.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 shutdown_connections 2024-04-25T18:43:49.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:49.148+0000 7fa431e00700 1 -- 172.21.15.63:0/552532085 wait complete. 2024-04-25T18:43:49.548 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:49 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/552532085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:49.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:49 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/552532085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:49.605 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:49 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/552532085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:43:50.076 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T18:43:50.077 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T18:43:50.077 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T18:43:50.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T18:43:50.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T18:43:50.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T18:43:50.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T18:43:50.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T18:43:50.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T18:43:50.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T18:43:50.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T18:43:50.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T18:43:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:50 smithi063 ceph-mon[28006]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:43:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:50 smithi070 ceph-mon[32820]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:43:50.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:50 smithi114 ceph-mon[32988]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:43:51.115 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:51.121 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:51.133 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:51.133 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:52.043 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:52.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:52 smithi063 ceph-mon[28006]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:52 smithi070 ceph-mon[32820]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:52 smithi114 ceph-mon[32988]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:43:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:54 smithi063 ceph-mon[28006]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T18:43:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:54 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-25T18:43:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:54 smithi070 ceph-mon[32820]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T18:43:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:54 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-25T18:43:54.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:54 smithi114 ceph-mon[32988]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:43:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-25T18:43:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:54 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]: dispatch 2024-04-25T18:43:55.514 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:55 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T18:43:55.514 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:55 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-25T18:43:55.514 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:55 smithi063 ceph-mon[28006]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T18:43:55.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:55 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T18:43:55.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:55 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-25T18:43:55.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:55 smithi070 ceph-mon[32820]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T18:43:55.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:55 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-25T18:43:55.822 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:55 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.29", "id": [3, 9]}]': finished 2024-04-25T18:43:55.822 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:55 smithi114 ceph-mon[32988]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T18:43:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:56.416 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:56 smithi063 ceph-mon[28006]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:43:56.417 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:56 smithi063 ceph-mon[28006]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T18:43:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:56 smithi070 ceph-mon[32820]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:43:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:56 smithi070 ceph-mon[32820]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T18:43:56.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:56 smithi114 ceph-mon[32988]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:43:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:56 smithi114 ceph-mon[32988]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T18:43:57.573 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:57.578 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.577+0000 7fe54ec00700 1 -- 172.21.15.63:0/216671801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 msgr2=0x7fe548108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.577+0000 7fe54ec00700 1 --2- 172.21.15.63:0/216671801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 0x7fe548108e10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe538009990 tx=0x7fe538009ca0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 -- 172.21.15.63:0/216671801 shutdown_connections 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 --2- 172.21.15.63:0/216671801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806b0f0 0x7fe54810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 --2- 172.21.15.63:0/216671801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 0x7fe548108e10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 --2- 172.21.15.63:0/216671801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806a750 0x7fe54806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 -- 172.21.15.63:0/216671801 >> 172.21.15.63:0/216671801 conn(0x7fe5480788f0 msgr2=0x7fe548078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:57.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 -- 172.21.15.63:0/216671801 shutdown_connections 2024-04-25T18:43:57.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.578+0000 7fe54ec00700 1 -- 172.21.15.63:0/216671801 wait complete. 2024-04-25T18:43:57.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.579+0000 7fe54ec00700 1 Processor -- start 2024-04-25T18:43:57.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 -- start start 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806a750 0x7fe5481991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806b0f0 0x7fe5481996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 0x7fe54819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112c00 con 0x7fe548108a30 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112a80 con 0x7fe54806b0f0 2024-04-25T18:43:57.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe54ec00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe548112d80 con 0x7fe54806a750 2024-04-25T18:43:57.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe547e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806a750 0x7fe5481991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:57.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.580+0000 7fe547e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806a750 0x7fe5481991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47588/0 (socket says 172.21.15.63:47588) 2024-04-25T18:43:57.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547e00700 1 -- 172.21.15.63:0/1060797821 learned_addr learned my addr 172.21.15.63:0/1060797821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:57.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe54cc00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 0x7fe54819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:57.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806b0f0 0x7fe5481996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 -- 172.21.15.63:0/1060797821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806a750 msgr2=0x7fe5481991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe54806a750 0x7fe5481991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 -- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 msgr2=0x7fe54819da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe548108a30 0x7fe54819da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.581+0000 7fe547400700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54819e1c0 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe547400700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806b0f0 0x7fe5481996f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe534010d00 tx=0x7fe53400b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53400bb50 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5340044d0 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534003e40 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54819e4b0 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.582+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe548101940 con 0x7fe54806b0f0 2024-04-25T18:43:57.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.583+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5480631d0 con 0x7fe54806b0f0 2024-04-25T18:43:57.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.585+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe534004000 con 0x7fe54806b0f0 2024-04-25T18:43:57.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.588+0000 7fe544c00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe52c0a3d50 0x7fe52c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:57.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.588+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fe5340d3bc0 con 0x7fe54806b0f0 2024-04-25T18:43:57.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.588+0000 7fe547e00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe52c0a3d50 0x7fe52c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:57.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.589+0000 7fe544c00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] conn(0x7fe52c0ad990 0x7fe52c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:57.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.589+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe52c0b0500 con 0x7fe52c0ad990 2024-04-25T18:43:57.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.589+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fe5340d82a0 con 0x7fe54806b0f0 2024-04-25T18:43:57.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.589+0000 7fe547e00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe52c0a3d50 0x7fe52c0a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe53800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:57.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.589+0000 7fe54cc00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] conn(0x7fe52c0ad990 0x7fe52c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:57.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.590+0000 7fe54cc00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] conn(0x7fe52c0ad990 0x7fe52c0afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:57.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.591+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== osd.7 v2:172.21.15.70:6816/568332364 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe52c0b0500 con 0x7fe52c0ad990 2024-04-25T18:43:57.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.627+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 --> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe5480635d0 con 0x7fe52c0ad990 2024-04-25T18:43:57.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.628+0000 7fe544c00700 1 -- 172.21.15.63:0/1060797821 <== osd.7 v2:172.21.15.70:6816/568332364 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe5480635d0 con 0x7fe52c0ad990 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.628+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] conn(0x7fe52c0ad990 msgr2=0x7fe52c0afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.628+0000 7fe54ec00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] conn(0x7fe52c0ad990 0x7fe52c0afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.628+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe52c0a3d50 msgr2=0x7fe52c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.628+0000 7fe54ec00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe52c0a3d50 0x7fe52c0a6210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe538006010 tx=0x7fe53800b280 comp rx=0 tx=0).stop 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806b0f0 msgr2=0x7fe5481996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 --2- 172.21.15.63:0/1060797821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe54806b0f0 0x7fe5481996f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe534010d00 tx=0x7fe53400b5a0 comp rx=0 tx=0).stop 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54cc00700 1 -- 172.21.15.63:0/1060797821 reap_dead start 2024-04-25T18:43:57.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 shutdown_connections 2024-04-25T18:43:57.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 >> 172.21.15.63:0/1060797821 conn(0x7fe5480788f0 msgr2=0x7fe5480ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:57.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 shutdown_connections 2024-04-25T18:43:57.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:57.629+0000 7fe54ec00700 1 -- 172.21.15.63:0/1060797821 wait complete. 2024-04-25T18:43:58.580 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:43:58 smithi063 ceph-mon[28006]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:43:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:43:58 smithi070 ceph-mon[32820]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:43:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:43:58 smithi114 ceph-mon[32988]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-25T18:43:59.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 -- 172.21.15.63:0/3336996962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f695410e3b0 msgr2=0x7f6954115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 --2- 172.21.15.63:0/3336996962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f695410e3b0 0x7f6954115f80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f694c00b0a0 tx=0x7f694c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 -- 172.21.15.63:0/3336996962 shutdown_connections 2024-04-25T18:43:59.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 --2- 172.21.15.63:0/3336996962 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 0x7f69541096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 --2- 172.21.15.63:0/3336996962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f695410e3b0 0x7f6954115f80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 --2- 172.21.15.63:0/3336996962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6954109c80 0x7f695410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 -- 172.21.15.63:0/3336996962 >> 172.21.15.63:0/3336996962 conn(0x7f6954076210 msgr2=0x7f6954107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 -- 172.21.15.63:0/3336996962 shutdown_connections 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.578+0000 7f6959600700 1 -- 172.21.15.63:0/3336996962 wait complete. 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 Processor -- start 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 -- start start 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 0x7f695419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 0x7f695419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 0x7f69541a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69541192a0 con 0x7f695410e3b0 2024-04-25T18:43:59.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6954119120 con 0x7f6954109c80 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6959600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6954119420 con 0x7f69541092d0 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6953400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 0x7f69541a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6952000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 0x7f695419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.580+0000 7f6952a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 0x7f695419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6953400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 0x7f69541a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56298/0 (socket says 172.21.15.63:56298) 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 0x7f695419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47614/0 (socket says 172.21.15.63:47614) 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 0x7f695419fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39952/0 (socket says 172.21.15.63:39952) 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6953400700 1 -- 172.21.15.63:0/171377426 learned_addr learned my addr 172.21.15.63:0/171377426 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 -- 172.21.15.63:0/171377426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 msgr2=0x7f695419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69541092d0 0x7f695419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 -- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 msgr2=0x7f69541a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 0x7f69541a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69541a4730 con 0x7f6954109c80 2024-04-25T18:43:59.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6953400700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695410e3b0 0x7f69541a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f6952000700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 0x7f695419fcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694000e9c0 tx=0x7f694000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.581+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694000c9c0 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.582+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f694000cb60 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.582+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940018680 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.582+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69541a4a20 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.582+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69541ac2a0 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.583+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69540631d0 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.585+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6940010430 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.586+0000 7f694b400700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69380a3d50 0x7f69380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.586+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f69400d0770 con 0x7f6954109c80 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.586+0000 7f6952a00700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69380a3d50 0x7f69380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.587+0000 7f694b400700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] conn(0x7f69380ad990 0x7f69380afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.587+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f694000be50 con 0x7f69380ad990 2024-04-25T18:43:59.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.587+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f69400d0b20 con 0x7f6954109c80 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.587+0000 7f6953400700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] conn(0x7f69380ad990 0x7f69380afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.587+0000 7f6952a00700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69380a3d50 0x7f69380a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6944009c60 tx=0x7f6944009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.588+0000 7f6953400700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] conn(0x7f69380ad990 0x7f69380afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.589+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== osd.6 v2:172.21.15.63:6818/3710200907 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f694000be50 con 0x7f69380ad990 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.607+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 --> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f69540635d0 con 0x7f69380ad990 2024-04-25T18:43:59.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.607+0000 7f694b400700 1 -- 172.21.15.63:0/171377426 <== osd.6 v2:172.21.15.63:6818/3710200907 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f69540635d0 con 0x7f69380ad990 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] conn(0x7f69380ad990 msgr2=0x7f69380afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] conn(0x7f69380ad990 0x7f69380afe30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69380a3d50 msgr2=0x7f69380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f69380a3d50 0x7f69380a6210 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6944009c60 tx=0x7f6944009260 comp rx=0 tx=0).stop 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 msgr2=0x7f695419fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6959600700 1 --2- 172.21.15.63:0/171377426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6954109c80 0x7f695419fcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f694000e9c0 tx=0x7f694000ecd0 comp rx=0 tx=0).stop 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.608+0000 7f6953400700 1 -- 172.21.15.63:0/171377426 reap_dead start 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.609+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 shutdown_connections 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.609+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 >> 172.21.15.63:0/171377426 conn(0x7f6954076210 msgr2=0x7f69540767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.609+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 shutdown_connections 2024-04-25T18:43:59.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:43:58.609+0000 7f6959600700 1 -- 172.21.15.63:0/171377426 wait complete. 2024-04-25T18:44:00.455 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:00 smithi063 ceph-mon[28006]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 -- 172.21.15.63:0/1510713657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 msgr2=0x7f29d4077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1510713657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 0x7f29d4077350 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009990 tx=0x7f29c4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 -- 172.21.15.63:0/1510713657 shutdown_connections 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1510713657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29d41137f0 0x7f29d4115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1510713657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 0x7f29d4077350 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1510713657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d41132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 -- 172.21.15.63:0/1510713657 >> 172.21.15.63:0/1510713657 conn(0x7f29d4100620 msgr2=0x7f29d4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 -- 172.21.15.63:0/1510713657 shutdown_connections 2024-04-25T18:44:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.483+0000 7f29d8a00700 1 -- 172.21.15.63:0/1510713657 wait complete. 2024-04-25T18:44:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.484+0000 7f29d8a00700 1 Processor -- start 2024-04-25T18:44:00.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.484+0000 7f29d8a00700 1 -- start start 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 0x7f29d419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29d41137f0 0x7f29d41a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4119180 con 0x7f29d4076f70 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4119000 con 0x7f29d4077890 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d8a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4119300 con 0x7f29d41137f0 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d1e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d1e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d419fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39970/0 (socket says 172.21.15.63:39970) 2024-04-25T18:44:00.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d2800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 0x7f29d419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:00.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d1e00700 1 -- 172.21.15.63:0/1644586450 learned_addr learned my addr 172.21.15.63:0/1644586450 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.485+0000 7f29d3200700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29d41137f0 0x7f29d41a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 -- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29d41137f0 msgr2=0x7f29d41a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f29d41137f0 0x7f29d41a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 -- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 msgr2=0x7f29d419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29d4076f70 0x7f29d419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d41a46a0 con 0x7f29d4077890 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d1e00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d419fbd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29c000e9a0 tx=0x7f29c000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c000ca10 con 0x7f29d4077890 2024-04-25T18:44:00.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29c000cbb0 con 0x7f29d4077890 2024-04-25T18:44:00.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0018680 con 0x7f29d4077890 2024-04-25T18:44:00.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d41a4990 con 0x7f29d4077890 2024-04-25T18:44:00.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.486+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d41ac210 con 0x7f29d4077890 2024-04-25T18:44:00.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.487+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f29d4063150 con 0x7f29d4077890 2024-04-25T18:44:00.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.489+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29c0010430 con 0x7f29d4077890 2024-04-25T18:44:00.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29cb400700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f29b80a3d50 0x7f29b80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:00.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f29c00d0c80 con 0x7f29d4077890 2024-04-25T18:44:00.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29d2800700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f29b80a3d50 0x7f29b80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:00.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29cb400700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] conn(0x7f29b80ad990 0x7f29b80afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:00.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29b80b0500 con 0x7f29b80ad990 2024-04-25T18:44:00.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.491+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f29c00d1030 con 0x7f29d4077890 2024-04-25T18:44:00.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.492+0000 7f29d3200700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] conn(0x7f29b80ad990 0x7f29b80afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:00.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.492+0000 7f29d2800700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f29b80a3d50 0x7f29b80a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b2e0 tx=0x7f29c4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:00.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.492+0000 7f29d3200700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] conn(0x7f29b80ad990 0x7f29b80afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:00.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.493+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== osd.11 v2:172.21.15.114:6824/3297229799 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29b80b0500 con 0x7f29b80ad990 2024-04-25T18:44:00.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.511+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 --> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f29d4063620 con 0x7f29b80ad990 2024-04-25T18:44:00.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.512+0000 7f29cb400700 1 -- 172.21.15.63:0/1644586450 <== osd.11 v2:172.21.15.114:6824/3297229799 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f29d4063620 con 0x7f29b80ad990 2024-04-25T18:44:00.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.512+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] conn(0x7f29b80ad990 msgr2=0x7f29b80afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.512+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] conn(0x7f29b80ad990 0x7f29b80afe30 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f29b80a3d50 msgr2=0x7f29b80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f29b80a3d50 0x7f29b80a6210 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f29c400b2e0 tx=0x7f29c4005960 comp rx=0 tx=0).stop 2024-04-25T18:44:00.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 msgr2=0x7f29d419fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 --2- 172.21.15.63:0/1644586450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29d4077890 0x7f29d419fbd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f29c000e9a0 tx=0x7f29c000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d3200700 1 -- 172.21.15.63:0/1644586450 reap_dead start 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 shutdown_connections 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 >> 172.21.15.63:0/1644586450 conn(0x7f29d4100620 msgr2=0x7f29d4102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 shutdown_connections 2024-04-25T18:44:00.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:00.513+0000 7f29d8a00700 1 -- 172.21.15.63:0/1644586450 wait complete. 2024-04-25T18:44:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:00 smithi070 ceph-mon[32820]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T18:44:00.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:00 smithi114 ceph-mon[32988]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-25T18:44:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:02 smithi070 ceph-mon[32820]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-25T18:44:02.728 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:02 smithi063 ceph-mon[28006]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-25T18:44:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:02 smithi114 ceph-mon[32988]: pgmap v390: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-04-25T18:44:03.018 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 -- 172.21.15.63:0/222288037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc109c80 msgr2=0x7fa1dc10dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/222288037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc109c80 0x7fa1dc10dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80099c0 tx=0x7fa1c8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 -- 172.21.15.63:0/222288037 shutdown_connections 2024-04-25T18:44:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/222288037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc10e3b0 0x7fa1dc115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/222288037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc1092d0 0x7fa1dc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/222288037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc109c80 0x7fa1dc10dcd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.427+0000 7fa1e0e00700 1 -- 172.21.15.63:0/222288037 >> 172.21.15.63:0/222288037 conn(0x7fa1dc076210 msgr2=0x7fa1dc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.428+0000 7fa1e0e00700 1 -- 172.21.15.63:0/222288037 shutdown_connections 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.428+0000 7fa1e0e00700 1 -- 172.21.15.63:0/222288037 wait complete. 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.429+0000 7fa1e0e00700 1 Processor -- start 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.429+0000 7fa1e0e00700 1 -- start start 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc1092d0 0x7fa1dc19f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc109c80 0x7fa1dc19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc10e3b0 0x7fa1dc1a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc119220 con 0x7fa1dc10e3b0 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc1190a0 con 0x7fa1dc109c80 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1e0e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc1193a0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1da800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc10e3b0 0x7fa1dc1a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:03.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1d9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc109c80 0x7fa1dc19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1d9e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc1092d0 0x7fa1dc19f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1d9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc109c80 0x7fa1dc19fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39998/0 (socket says 172.21.15.63:39998) 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.430+0000 7fa1d9400700 1 -- 172.21.15.63:0/1086257207 learned_addr learned my addr 172.21.15.63:0/1086257207 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 -- 172.21.15.63:0/1086257207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc109c80 msgr2=0x7fa1dc19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1dc109c80 0x7fa1dc19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 -- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc10e3b0 msgr2=0x7fa1dc1a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc10e3b0 0x7fa1dc1a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc1a46c0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d9e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc1092d0 0x7fa1dc19f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00e9f0 tx=0x7fa1cc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.431+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc00c9c0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.432+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc1a49b0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.432+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1cc00cb60 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.432+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc018680 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.432+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc1ac230 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.433+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa1dc0631d0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.435+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa1cc010430 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.437+0000 7fa1d2a00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa1c00a3d50 0x7fa1c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.437+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fa1cc0d07d0 con 0x7fa1dc1092d0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.437+0000 7fa1d9400700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa1c00a3d50 0x7fa1c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.438+0000 7fa1d2a00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] conn(0x7fa1c00ad990 0x7fa1c00afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.438+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1cc004500 con 0x7fa1c00ad990 2024-04-25T18:44:03.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.438+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fa1cc0d0b80 con 0x7fa1dc1092d0 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.438+0000 7fa1da800700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] conn(0x7fa1c00ad990 0x7fa1c00afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.438+0000 7fa1d9400700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa1c00a3d50 0x7fa1c00a6210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8003760 tx=0x7fa1c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.439+0000 7fa1da800700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] conn(0x7fa1c00ad990 0x7fa1c00afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.440+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== osd.0 v2:172.21.15.63:6802/237733214 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1cc004500 con 0x7fa1c00ad990 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.458+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 --> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa1dc063600 con 0x7fa1c00ad990 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.459+0000 7fa1d2a00700 1 -- 172.21.15.63:0/1086257207 <== osd.0 v2:172.21.15.63:6802/237733214 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa1dc063600 con 0x7fa1c00ad990 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] conn(0x7fa1c00ad990 msgr2=0x7fa1c00afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] conn(0x7fa1c00ad990 0x7fa1c00afe30 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa1c00a3d50 msgr2=0x7fa1c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa1c00a3d50 0x7fa1c00a6210 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8003760 tx=0x7fa1c8005fb0 comp rx=0 tx=0).stop 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc1092d0 msgr2=0x7fa1dc19f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 --2- 172.21.15.63:0/1086257207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa1dc1092d0 0x7fa1dc19f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00e9f0 tx=0x7fa1cc00ed00 comp rx=0 tx=0).stop 2024-04-25T18:44:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1da800700 1 -- 172.21.15.63:0/1086257207 reap_dead start 2024-04-25T18:44:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 shutdown_connections 2024-04-25T18:44:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 >> 172.21.15.63:0/1086257207 conn(0x7fa1dc076210 msgr2=0x7fa1dc0767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.460+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 shutdown_connections 2024-04-25T18:44:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:03.461+0000 7fa1e0e00700 1 -- 172.21.15.63:0/1086257207 wait complete. 2024-04-25T18:44:04.657 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:04 smithi063 ceph-mon[28006]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-04-25T18:44:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:04 smithi070 ceph-mon[32820]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-04-25T18:44:04.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:04 smithi114 ceph-mon[32988]: pgmap v391: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-04-25T18:44:05.736 INFO:teuthology.orchestra.run.smithi063.stdout:210453397569 2024-04-25T18:44:05.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T18:44:06.644 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:06 smithi063 ceph-mon[28006]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:44:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:06 smithi070 ceph-mon[32820]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:44:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:06 smithi114 ceph-mon[32988]: pgmap v392: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-25T18:44:07.667 INFO:teuthology.orchestra.run.smithi063.stdout:360777252901 2024-04-25T18:44:07.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T18:44:08.674 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:08 smithi063 ceph-mon[28006]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:08 smithi070 ceph-mon[32820]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:08 smithi114 ceph-mon[32988]: pgmap v393: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:09.017 INFO:teuthology.orchestra.run.smithi063.stdout:34359738470 2024-04-25T18:44:09.017 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T18:44:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 -- 172.21.15.63:0/149303371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc075590 msgr2=0x7fc9bc10ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 --2- 172.21.15.63:0/149303371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc075590 0x7fc9bc10ef00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009990 tx=0x7fc9a8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 -- 172.21.15.63:0/149303371 shutdown_connections 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 --2- 172.21.15.63:0/149303371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc074c70 0x7fc9bc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 --2- 172.21.15.63:0/149303371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc10f440 0x7fc9bc111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 --2- 172.21.15.63:0/149303371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc075590 0x7fc9bc10ef00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 -- 172.21.15.63:0/149303371 >> 172.21.15.63:0/149303371 conn(0x7fc9bc079ba0 msgr2=0x7fc9bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 -- 172.21.15.63:0/149303371 shutdown_connections 2024-04-25T18:44:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.252+0000 7fc9c1600700 1 -- 172.21.15.63:0/149303371 wait complete. 2024-04-25T18:44:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.254+0000 7fc9c1600700 1 Processor -- start 2024-04-25T18:44:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.254+0000 7fc9c1600700 1 -- start start 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.254+0000 7fc9c1600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc074c70 0x7fc9bc19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9c1600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc075590 0x7fc9bc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9c1600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 0x7fc9bc19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9c1600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc114e50 con 0x7fc9bc074c70 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9c1600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc114cd0 con 0x7fc9bc075590 2024-04-25T18:44:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9c1600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc114fd0 con 0x7fc9bc10f440 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9bb400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 0x7fc9bc19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9ba000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc075590 0x7fc9bc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9bb400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 0x7fc9bc19fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40716/0 (socket says 172.21.15.63:40716) 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9baa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc074c70 0x7fc9bc19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.255+0000 7fc9bb400700 1 -- 172.21.15.63:0/2610540499 learned_addr learned my addr 172.21.15.63:0/2610540499 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 -- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 msgr2=0x7fc9bc19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 0x7fc9bc19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 -- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc074c70 msgr2=0x7fc9bc19b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9bc074c70 0x7fc9bc19b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc1a0380 con 0x7fc9bc075590 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9bb400700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc9bc10f440 0x7fc9bc19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9ba000700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc075590 0x7fc9bc19b900 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8000c00 tx=0x7fc9a80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8026070 con 0x7fc9bc075590 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9a8021410 con 0x7fc9bc075590 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9bc1a0610 con 0x7fc9bc075590 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.256+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8017650 con 0x7fc9bc075590 2024-04-25T18:44:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.257+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9bc1a7ef0 con 0x7fc9bc075590 2024-04-25T18:44:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.258+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9bc0631d0 con 0x7fc9bc075590 2024-04-25T18:44:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.260+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc9a8029070 con 0x7fc9bc075590 2024-04-25T18:44:09.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.261+0000 7fc9b3400700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc9a00a3cd0 0x7fc9a00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.261+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fc9a80d12a0 con 0x7fc9bc075590 2024-04-25T18:44:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.261+0000 7fc9baa00700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc9a00a3cd0 0x7fc9a00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.262+0000 7fc9b3400700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] conn(0x7fc9a00ad910 0x7fc9a00afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.262+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9a00b0480 con 0x7fc9a00ad910 2024-04-25T18:44:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.262+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fc9a80d8050 con 0x7fc9bc075590 2024-04-25T18:44:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.262+0000 7fc9baa00700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc9a00a3cd0 0x7fc9a00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c60 tx=0x7fc9ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.262+0000 7fc9bb400700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] conn(0x7fc9a00ad910 0x7fc9a00afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.263+0000 7fc9bb400700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] conn(0x7fc9a00ad910 0x7fc9a00afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.264+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== osd.2 v2:172.21.15.114:6800/3090454830 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9a00b0480 con 0x7fc9a00ad910 2024-04-25T18:44:09.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.282+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 --> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc9bc063650 con 0x7fc9a00ad910 2024-04-25T18:44:09.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9b3400700 1 -- 172.21.15.63:0/2610540499 <== osd.2 v2:172.21.15.114:6800/3090454830 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc9bc063650 con 0x7fc9a00ad910 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] conn(0x7fc9a00ad910 msgr2=0x7fc9a00afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] conn(0x7fc9a00ad910 0x7fc9a00afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc9a00a3cd0 msgr2=0x7fc9a00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc9a00a3cd0 0x7fc9a00a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c60 tx=0x7fc9ac009260 comp rx=0 tx=0).stop 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc075590 msgr2=0x7fc9bc19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9c1600700 1 --2- 172.21.15.63:0/2610540499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc9bc075590 0x7fc9bc19b900 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8000c00 tx=0x7fc9a80047b0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.283+0000 7fc9bb400700 1 -- 172.21.15.63:0/2610540499 reap_dead start 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.284+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 shutdown_connections 2024-04-25T18:44:09.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.284+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 >> 172.21.15.63:0/2610540499 conn(0x7fc9bc079ba0 msgr2=0x7fc9bc0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:09.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.284+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 shutdown_connections 2024-04-25T18:44:09.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.284+0000 7fc9c1600700 1 -- 172.21.15.63:0/2610540499 wait complete. 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 -- 172.21.15.63:0/3932322466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 msgr2=0x7f3d50111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 --2- 172.21.15.63:0/3932322466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 0x7f3d50111830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b0a0 tx=0x7f3d4800b3b0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 -- 172.21.15.63:0/3932322466 shutdown_connections 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 --2- 172.21.15.63:0/3932322466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 0x7f3d50111830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 --2- 172.21.15.63:0/3932322466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d50075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 --2- 172.21.15.63:0/3932322466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d50075590 0x7f3d5010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 -- 172.21.15.63:0/3932322466 >> 172.21.15.63:0/3932322466 conn(0x7f3d50079ba0 msgr2=0x7f3d50079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 -- 172.21.15.63:0/3932322466 shutdown_connections 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.823+0000 7f3d54c00700 1 -- 172.21.15.63:0/3932322466 wait complete. 2024-04-25T18:44:09.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.824+0000 7f3d54c00700 1 Processor -- start 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.824+0000 7f3d54c00700 1 -- start start 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d5019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d50075590 0x7f3d5019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 0x7f3d5019fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3d50114e40 con 0x7f3d50074c70 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d50114cc0 con 0x7f3d50075590 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d54c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d50114fc0 con 0x7f3d5010f440 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d4de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d5019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d4de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d5019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60940/0 (socket says 172.21.15.63:60940) 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d4de00700 1 -- 172.21.15.63:0/1352660498 learned_addr learned my addr 172.21.15.63:0/1352660498 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d4e800700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 0x7f3d5019fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.825+0000 7f3d4d400700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d50075590 0x7f3d5019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 -- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 msgr2=0x7f3d5019fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d5010f440 0x7f3d5019fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 -- 172.21.15.63:0/1352660498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d50075590 msgr2=0x7f3d5019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d50075590 0x7f3d5019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d501a0290 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d4de00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d5019b2d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e9a0 tx=0x7f3d4000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4000ca10 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.826+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d501a0580 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.827+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d4000cbb0 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.827+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d40018680 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.827+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d501a7e00 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.827+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d50063150 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.831+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3d40010430 con 0x7f3d50074c70 2024-04-25T18:44:09.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d46a00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3d340a3cd0 0x7f3d340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f3d400d0cd0 con 0x7f3d50074c70 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d4d400700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3d340a3cd0 0x7f3d340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d46a00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] conn(0x7f3d340ad910 0x7f3d340afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d4000be50 con 0x7f3d340ad910 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.832+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f3d400d1120 con 0x7f3d50074c70 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.833+0000 7f3d4e800700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] conn(0x7f3d340ad910 0x7f3d340afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:09.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.833+0000 7f3d4d400700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3d340a3cd0 0x7f3d340a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3d50111160 tx=0x7f3d3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.833+0000 7f3d4e800700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] conn(0x7f3d340ad910 0x7f3d340afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.834+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== osd.5 v2:172.21.15.114:6808/4152604571 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d4000be50 con 0x7f3d340ad910 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.853+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 --> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d50063550 con 0x7f3d340ad910 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.853+0000 7f3d46a00700 1 -- 172.21.15.63:0/1352660498 <== osd.5 v2:172.21.15.114:6808/4152604571 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3d50063550 con 0x7f3d340ad910 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] conn(0x7f3d340ad910 msgr2=0x7f3d340afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] conn(0x7f3d340ad910 0x7f3d340afdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3d340a3cd0 msgr2=0x7f3d340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3d340a3cd0 0x7f3d340a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3d50111160 tx=0x7f3d3c009260 comp rx=0 tx=0).stop 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 msgr2=0x7f3d5019b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 --2- 172.21.15.63:0/1352660498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d50074c70 0x7f3d5019b2d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e9a0 tx=0x7f3d4000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d4e800700 1 -- 172.21.15.63:0/1352660498 reap_dead start 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 shutdown_connections 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.854+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 >> 172.21.15.63:0/1352660498 conn(0x7f3d50079ba0 msgr2=0x7f3d500ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.855+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 shutdown_connections 2024-04-25T18:44:09.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:09.855+0000 7f3d54c00700 1 -- 172.21.15.63:0/1352660498 wait complete. 2024-04-25T18:44:10.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:10 smithi063 ceph-mon[28006]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:10 smithi070 ceph-mon[32820]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:10 smithi114 ceph-mon[32988]: pgmap v394: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:11.312 INFO:teuthology.orchestra.run.smithi063.stdout:231928234043 2024-04-25T18:44:11.313 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T18:44:11.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 -- 172.21.15.63:0/2657818442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb350074c70 msgr2=0x7fb350075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 --2- 172.21.15.63:0/2657818442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb350074c70 0x7fb350075050 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098f0 tx=0x7fb340009c00 comp rx=0 tx=0).stop 2024-04-25T18:44:11.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 -- 172.21.15.63:0/2657818442 shutdown_connections 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 --2- 172.21.15.63:0/2657818442 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb35010f440 0x7fb350111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 --2- 172.21.15.63:0/2657818442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb350074c70 0x7fb350075050 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 --2- 172.21.15.63:0/2657818442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 -- 172.21.15.63:0/2657818442 >> 172.21.15.63:0/2657818442 conn(0x7fb350079bf0 msgr2=0x7fb35007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 -- 172.21.15.63:0/2657818442 shutdown_connections 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.583+0000 7fb354a00700 1 -- 172.21.15.63:0/2657818442 wait complete. 2024-04-25T18:44:11.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 Processor -- start 2024-04-25T18:44:11.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 -- start start 2024-04-25T18:44:11.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb350074c70 0x7fb35019b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35010f440 0x7fb35019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb350114ed0 con 0x7fb350075590 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb350114d50 con 0x7fb35010f440 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.585+0000 7fb354a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb350115050 con 0x7fb350074c70 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60962/0 (socket says 172.21.15.63:60962) 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 -- 172.21.15.63:0/1486274701 learned_addr learned my addr 172.21.15.63:0/1486274701 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34de00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb350074c70 0x7fb35019b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:11.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34e800700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35010f440 0x7fb35019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 -- 172.21.15.63:0/1486274701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb350074c70 msgr2=0x7fb35019b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb350074c70 0x7fb35019b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 -- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35010f440 msgr2=0x7fb35019fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35010f440 0x7fb35019fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34d400700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3501a0420 con 0x7fb350075590 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34de00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb350074c70 0x7fb35019b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:11.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.586+0000 7fb34e800700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb35010f440 0x7fb35019fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb34d400700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35019b9a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00e9c0 tx=0x7fb33c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c004a30 con 0x7fb350075590 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3501a0710 con 0x7fb350075590 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3501a7f90 con 0x7fb350075590 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb33c00cc70 con 0x7fb350075590 2024-04-25T18:44:11.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.587+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0054a0 con 0x7fb350075590 2024-04-25T18:44:11.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.589+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb330001030 con 0x7fb350075590 2024-04-25T18:44:11.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.591+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb33c004bd0 con 0x7fb350075590 2024-04-25T18:44:11.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb346a00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb3340a3d50 0x7fb3340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:11.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fb33c0d0590 con 0x7fb350075590 2024-04-25T18:44:11.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb34de00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb3340a3d50 0x7fb3340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:11.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb346a00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] conn(0x7fb3340ad990 0x7fb3340afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:11.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3340b0500 con 0x7fb3340ad990 2024-04-25T18:44:11.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.593+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fb33c0d7050 con 0x7fb350075590 2024-04-25T18:44:11.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.594+0000 7fb34e800700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] conn(0x7fb3340ad990 0x7fb3340afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:11.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.594+0000 7fb34de00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb3340a3d50 0x7fb3340a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098c0 tx=0x7fb340019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:11.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.594+0000 7fb34e800700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] conn(0x7fb3340ad990 0x7fb3340afe30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:11.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.595+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== osd.1 v2:172.21.15.70:6800/135943968 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3340b0500 con 0x7fb3340ad990 2024-04-25T18:44:11.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.620+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 --> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb330002e60 con 0x7fb3340ad990 2024-04-25T18:44:11.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.620+0000 7fb346a00700 1 -- 172.21.15.63:0/1486274701 <== osd.1 v2:172.21.15.70:6800/135943968 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb330002e60 con 0x7fb3340ad990 2024-04-25T18:44:11.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] conn(0x7fb3340ad990 msgr2=0x7fb3340afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] conn(0x7fb3340ad990 0x7fb3340afe30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb3340a3d50 msgr2=0x7fb3340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb3340a3d50 0x7fb3340a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098c0 tx=0x7fb340019040 comp rx=0 tx=0).stop 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 msgr2=0x7fb35019b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 --2- 172.21.15.63:0/1486274701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb350075590 0x7fb35019b9a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb33c00e9c0 tx=0x7fb33c00ecd0 comp rx=0 tx=0).stop 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb34e800700 1 -- 172.21.15.63:0/1486274701 reap_dead start 2024-04-25T18:44:11.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 shutdown_connections 2024-04-25T18:44:11.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.621+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 >> 172.21.15.63:0/1486274701 conn(0x7fb350079bf0 msgr2=0x7fb3500ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:11.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.622+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 shutdown_connections 2024-04-25T18:44:11.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:11.622+0000 7fb354a00700 1 -- 172.21.15.63:0/1486274701 wait complete. 2024-04-25T18:44:12.607 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:12 smithi063 ceph-mon[28006]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T18:44:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:12 smithi070 ceph-mon[32820]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T18:44:12.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:12 smithi114 ceph-mon[32988]: pgmap v395: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-25T18:44:13.774 INFO:teuthology.orchestra.run.smithi063.stdout:77309411420 2024-04-25T18:44:13.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T18:44:14.353 INFO:teuthology.orchestra.run.smithi063.stdout:55834574943 2024-04-25T18:44:14.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T18:44:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:14 smithi063 ceph-mon[28006]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:14 smithi070 ceph-mon[32820]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:14 smithi114 ceph-mon[32988]: pgmap v396: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:14.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 -- 172.21.15.63:0/2617062497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 msgr2=0x7f5248077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 --2- 172.21.15.63:0/2617062497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f5248077350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f523c009990 tx=0x7f523c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 -- 172.21.15.63:0/2617062497 shutdown_connections 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 --2- 172.21.15.63:0/2617062497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f5248077350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 --2- 172.21.15.63:0/2617062497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5248113920 0x7f5248115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 --2- 172.21.15.63:0/2617062497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5248077890 0x7f52481133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 -- 172.21.15.63:0/2617062497 >> 172.21.15.63:0/2617062497 conn(0x7f5248100770 msgr2=0x7f5248102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 -- 172.21.15.63:0/2617062497 shutdown_connections 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.815+0000 7f5250400700 1 -- 172.21.15.63:0/2617062497 wait complete. 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 Processor -- start 2024-04-25T18:44:14.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 -- start start 2024-04-25T18:44:14.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f52481a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5248077890 0x7f52481a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5248113920 0x7f524819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5248119310 con 0x7f5248113920 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5248119190 con 0x7f5248077890 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.817+0000 7f5250400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5248119490 con 0x7f5248076f70 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f52481a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f52481a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40784/0 (socket says 172.21.15.63:40784) 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 -- 172.21.15.63:0/998751644 learned_addr learned my addr 172.21.15.63:0/998751644 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524e400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5248113920 0x7f524819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524d000700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5248077890 0x7f52481a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 -- 172.21.15.63:0/998751644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5248077890 msgr2=0x7f52481a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5248077890 0x7f52481a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 -- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5248113920 msgr2=0x7f524819f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5248113920 0x7f524819f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f524819fd90 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.818+0000 7f524da00700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f52481a5420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f523c005910 tx=0x7f523c00b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.819+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c027070 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.819+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52481a0020 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.819+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f523c022470 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.819+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523c017650 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.819+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52481ac2f0 con 0x7f5248076f70 2024-04-25T18:44:14.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.820+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5248063190 con 0x7f5248076f70 2024-04-25T18:44:14.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.822+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f523c02a070 con 0x7f5248076f70 2024-04-25T18:44:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.823+0000 7f5246000700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f52300a3d50 0x7f52300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.823+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f523c0d17a0 con 0x7f5248076f70 2024-04-25T18:44:14.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f524d000700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f52300a3d50 0x7f52300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f5246000700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] conn(0x7f52300ad990 0x7f52300afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f523c032ea0 con 0x7f52300ad990 2024-04-25T18:44:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f523c0d1b50 con 0x7f5248076f70 2024-04-25T18:44:14.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f524e400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] conn(0x7f52300ad990 0x7f52300afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.824+0000 7f524d000700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f52300a3d50 0x7f52300a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5248104320 tx=0x7f523400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:14.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.825+0000 7f524e400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] conn(0x7f52300ad990 0x7f52300afe30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:14.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.826+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== osd.8 v2:172.21.15.114:6816/1230878989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f523c032ea0 con 0x7f52300ad990 2024-04-25T18:44:14.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.845+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 --> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5248063590 con 0x7f52300ad990 2024-04-25T18:44:14.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.845+0000 7f5246000700 1 -- 172.21.15.63:0/998751644 <== osd.8 v2:172.21.15.114:6816/1230878989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5248063590 con 0x7f52300ad990 2024-04-25T18:44:14.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] conn(0x7f52300ad990 msgr2=0x7f52300afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] conn(0x7f52300ad990 0x7f52300afe30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f52300a3d50 msgr2=0x7f52300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f52300a3d50 0x7f52300a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5248104320 tx=0x7f523400b3c0 comp rx=0 tx=0).stop 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 msgr2=0x7f52481a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 --2- 172.21.15.63:0/998751644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5248076f70 0x7f52481a5420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f523c005910 tx=0x7f523c00b670 comp rx=0 tx=0).stop 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f524e400700 1 -- 172.21.15.63:0/998751644 reap_dead start 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 shutdown_connections 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.846+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 >> 172.21.15.63:0/998751644 conn(0x7f5248100770 msgr2=0x7f5248102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.847+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 shutdown_connections 2024-04-25T18:44:14.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:14.847+0000 7f5250400700 1 -- 172.21.15.63:0/998751644 wait complete. 2024-04-25T18:44:15.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 -- 172.21.15.63:0/1151187176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 msgr2=0x7f226810f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 --2- 172.21.15.63:0/1151187176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f226810f920 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f226000b0a0 tx=0x7f226000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 -- 172.21.15.63:0/1151187176 shutdown_connections 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 --2- 172.21.15.63:0/1151187176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 --2- 172.21.15.63:0/1151187176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f226810f920 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 --2- 172.21.15.63:0/1151187176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f226806b110 0x7f226806b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 -- 172.21.15.63:0/1151187176 >> 172.21.15.63:0/1151187176 conn(0x7f22680788f0 msgr2=0x7f2268078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 -- 172.21.15.63:0/1151187176 shutdown_connections 2024-04-25T18:44:15.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.954+0000 7f226f000700 1 -- 172.21.15.63:0/1151187176 wait complete. 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.955+0000 7f226f000700 1 Processor -- start 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 -- start start 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f226806b110 0x7f2268101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f2268102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2268112c00 con 0x7f226806bab0 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2268112a80 con 0x7f226806b110 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226f000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2268112d80 con 0x7f2268108a50 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f2267400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f2268102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f2267400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f2268102330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:32768/0 (socket says 172.21.15.63:32768) 2024-04-25T18:44:15.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f2267400700 1 -- 172.21.15.63:0/3257861078 learned_addr learned my addr 172.21.15.63:0/3257861078 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f226d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268102870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40788/0 (socket says 172.21.15.63:40788) 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.956+0000 7f2267e00700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f226806b110 0x7f2268101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 -- 172.21.15.63:0/3257861078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f226806b110 msgr2=0x7f2268101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f226806b110 0x7f2268101df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 -- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 msgr2=0x7f2268102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f226806bab0 0x7f2268102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22681a7680 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f226d000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268102870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f226000d010 tx=0x7f2260004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.957+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260021070 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.958+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22681a7910 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.958+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2260007e30 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.958+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600045a0 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.958+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22681a7e10 con 0x7f2268108a50 2024-04-25T18:44:15.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.958+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2268063110 con 0x7f2268108a50 2024-04-25T18:44:15.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.961+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2260004740 con 0x7f2268108a50 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.962+0000 7f2264c00700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f224c0a3cd0 0x7f224c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.962+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f226000f020 con 0x7f2268108a50 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2267e00700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f224c0a3cd0 0x7f224c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2264c00700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] conn(0x7f224c0ad910 0x7f224c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f224c0b0480 con 0x7f224c0ad910 2024-04-25T18:44:15.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f22600d1ee0 con 0x7f2268108a50 2024-04-25T18:44:15.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2267400700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] conn(0x7f224c0ad910 0x7f224c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:15.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.963+0000 7f2267e00700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f224c0a3cd0 0x7f224c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2258009be0 tx=0x7f2258009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:15.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.964+0000 7f2267400700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] conn(0x7f224c0ad910 0x7f224c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:15.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.965+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== osd.3 v2:172.21.15.63:6810/3120222680 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f224c0b0480 con 0x7f224c0ad910 2024-04-25T18:44:15.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.983+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 --> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2268063510 con 0x7f224c0ad910 2024-04-25T18:44:15.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f2264c00700 1 -- 172.21.15.63:0/3257861078 <== osd.3 v2:172.21.15.63:6810/3120222680 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2268063510 con 0x7f224c0ad910 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] conn(0x7f224c0ad910 msgr2=0x7f224c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] conn(0x7f224c0ad910 0x7f224c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f224c0a3cd0 msgr2=0x7f224c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f224c0a3cd0 0x7f224c0a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2258009be0 tx=0x7f2258009260 comp rx=0 tx=0).stop 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 msgr2=0x7f2268102870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.984+0000 7f226f000700 1 --2- 172.21.15.63:0/3257861078 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2268108a50 0x7f2268102870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f226000d010 tx=0x7f2260004d00 comp rx=0 tx=0).stop 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.985+0000 7f226d000700 1 -- 172.21.15.63:0/3257861078 reap_dead start 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.985+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 shutdown_connections 2024-04-25T18:44:15.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.985+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 >> 172.21.15.63:0/3257861078 conn(0x7f22680788f0 msgr2=0x7f22680ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:15.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.985+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 shutdown_connections 2024-04-25T18:44:15.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:15.985+0000 7f226f000700 1 -- 172.21.15.63:0/3257861078 wait complete. 2024-04-25T18:44:16.552 INFO:teuthology.orchestra.run.smithi063.stdout:176093659211 2024-04-25T18:44:16.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T18:44:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:16 smithi070 ceph-mon[32820]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:16 smithi114 ceph-mon[32988]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:16.846 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:16 smithi063 ceph-mon[28006]: pgmap v397: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 -- 172.21.15.63:0/4129300031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48104ee0 msgr2=0x7f2f481052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/4129300031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48104ee0 0x7f2f481052c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2f380099c0 tx=0x7f2f38009cd0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 -- 172.21.15.63:0/4129300031 shutdown_connections 2024-04-25T18:44:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/4129300031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48104ee0 0x7f2f481052c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/4129300031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f48109fc0 0x7f2f48111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/4129300031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48105890 0x7f2f481098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 -- 172.21.15.63:0/4129300031 >> 172.21.15.63:0/4129300031 conn(0x7f2f48100770 msgr2=0x7f2f48102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:17.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 -- 172.21.15.63:0/4129300031 shutdown_connections 2024-04-25T18:44:17.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.242+0000 7f2f4ca00700 1 -- 172.21.15.63:0/4129300031 wait complete. 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.243+0000 7f2f4ca00700 1 Processor -- start 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.243+0000 7f2f4ca00700 1 -- start start 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 0x7f2f4819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f48105890 0x7f2f4819b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48109fc0 0x7f2f4819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48114ed0 con 0x7f2f48105890 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48114d50 con 0x7f2f48104ee0 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f4ca00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48115050 con 0x7f2f48109fc0 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f45e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 0x7f2f4819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:17.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f45400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f48105890 0x7f2f4819b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f45e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 0x7f2f4819b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35980/0 (socket says 172.21.15.63:35980) 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f45e00700 1 -- 172.21.15.63:0/2259275530 learned_addr learned my addr 172.21.15.63:0/2259275530 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.244+0000 7f2f46800700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48109fc0 0x7f2f4819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 -- 172.21.15.63:0/2259275530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48109fc0 msgr2=0x7f2f4819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2f48109fc0 0x7f2f4819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 -- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f48105890 msgr2=0x7f2f4819b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f48105890 0x7f2f4819b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f481a03c0 con 0x7f2f48104ee0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f45e00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 0x7f2f4819b400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f38006010 tx=0x7f2f38005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38026070 con 0x7f2f48104ee0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f38003c70 con 0x7f2f48104ee0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f481a0650 con 0x7f2f48104ee0 2024-04-25T18:44:17.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800fb40 con 0x7f2f48104ee0 2024-04-25T18:44:17.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.245+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f481a7f30 con 0x7f2f48104ee0 2024-04-25T18:44:17.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.246+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2f48063150 con 0x7f2f48104ee0 2024-04-25T18:44:17.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.249+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2f38029070 con 0x7f2f48104ee0 2024-04-25T18:44:17.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.250+0000 7f2f3ea00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2f2c0a3cd0 0x7f2f2c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:17.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.250+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f2f380d15c0 con 0x7f2f48104ee0 2024-04-25T18:44:17.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.250+0000 7f2f45400700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2f2c0a3cd0 0x7f2f2c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.250+0000 7f2f3ea00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] conn(0x7f2f2c0ad910 0x7f2f2c0afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.250+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f38021e50 con 0x7f2f2c0ad910 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.251+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f2f380d1970 con 0x7f2f48104ee0 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.251+0000 7f2f46800700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] conn(0x7f2f2c0ad910 0x7f2f2c0afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.251+0000 7f2f45400700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2f2c0a3cd0 0x7f2f2c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f48102e20 tx=0x7f2f34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:17.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.251+0000 7f2f46800700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] conn(0x7f2f2c0ad910 0x7f2f2c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:17.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.252+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== osd.9 v2:172.21.15.63:6826/2032604816 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2f38021e50 con 0x7f2f2c0ad910 2024-04-25T18:44:17.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.271+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 --> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2f480635d0 con 0x7f2f2c0ad910 2024-04-25T18:44:17.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.271+0000 7f2f3ea00700 1 -- 172.21.15.63:0/2259275530 <== osd.9 v2:172.21.15.63:6826/2032604816 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2f480635d0 con 0x7f2f2c0ad910 2024-04-25T18:44:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] conn(0x7f2f2c0ad910 msgr2=0x7f2f2c0afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] conn(0x7f2f2c0ad910 0x7f2f2c0afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2f2c0a3cd0 msgr2=0x7f2f2c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f2f2c0a3cd0 0x7f2f2c0a6190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f48102e20 tx=0x7f2f34009260 comp rx=0 tx=0).stop 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 msgr2=0x7f2f4819b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 --2- 172.21.15.63:0/2259275530 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2f48104ee0 0x7f2f4819b400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f38006010 tx=0x7f2f38005280 comp rx=0 tx=0).stop 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f46800700 1 -- 172.21.15.63:0/2259275530 reap_dead start 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 shutdown_connections 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 >> 172.21.15.63:0/2259275530 conn(0x7f2f48100770 msgr2=0x7f2f48109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.272+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 shutdown_connections 2024-04-25T18:44:17.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:17.273+0000 7f2f4ca00700 1 -- 172.21.15.63:0/2259275530 wait complete. 2024-04-25T18:44:18.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 -- 172.21.15.63:0/4268050250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 msgr2=0x7f006410f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 --2- 172.21.15.63:0/4268050250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f006410f820 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f005c009f90 tx=0x7f005c00b330 comp rx=0 tx=0).stop 2024-04-25T18:44:18.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 -- 172.21.15.63:0/4268050250 shutdown_connections 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 --2- 172.21.15.63:0/4268050250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f006410f820 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 --2- 172.21.15.63:0/4268050250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064108950 0x7f0064108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 --2- 172.21.15.63:0/4268050250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064102950 0x7f0064102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 -- 172.21.15.63:0/4268050250 >> 172.21.15.63:0/4268050250 conn(0x7f00640fe420 msgr2=0x7f0064100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 -- 172.21.15.63:0/4268050250 shutdown_connections 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.574+0000 7f0069200700 1 -- 172.21.15.63:0/4268050250 wait complete. 2024-04-25T18:44:18.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.575+0000 7f0069200700 1 Processor -- start 2024-04-25T18:44:18.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 -- start start 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 0x7f0064074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f0064075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064108950 0x7f00640757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0064112b80 con 0x7f0064102950 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0064112a00 con 0x7f0064108950 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.576+0000 7f0069200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0064112d00 con 0x7f0064103310 2024-04-25T18:44:18.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 0x7f0064074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f0063400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f0064075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 0x7f0064074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58688/0 (socket says 172.21.15.63:58688) 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 -- 172.21.15.63:0/4261001772 learned_addr learned my addr 172.21.15.63:0/4261001772 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f0063e00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064108950 0x7f00640757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 -- 172.21.15.63:0/4261001772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 msgr2=0x7f0064075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f0064075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 -- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064108950 msgr2=0x7f00640757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064108950 0x7f00640757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00641ab9a0 con 0x7f0064102950 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f0063400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0064103310 0x7f0064075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:18.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f0063e00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0064108950 0x7f00640757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.577+0000 7f005b400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 0x7f0064074d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f005400e9c0 tx=0x7f005400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.578+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054004a30 con 0x7f0064102950 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.578+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f005400cc70 con 0x7f0064102950 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.578+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00641abc30 con 0x7f0064102950 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.578+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0054005490 con 0x7f0064102950 2024-04-25T18:44:18.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.578+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00641ac1c0 con 0x7f0064102950 2024-04-25T18:44:18.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.579+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0064063110 con 0x7f0064102950 2024-04-25T18:44:18.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.581+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0054004bd0 con 0x7f0064102950 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.582+0000 7f0060c00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f00480a3d50 0x7f00480a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.583+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f00540d0a60 con 0x7f0064102950 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.583+0000 7f0063400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f00480a3d50 0x7f00480a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.583+0000 7f0060c00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] conn(0x7f00480ad990 0x7f00480afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.583+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f005401ee50 con 0x7f00480ad990 2024-04-25T18:44:18.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.583+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f00540d0e10 con 0x7f0064102950 2024-04-25T18:44:18.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.584+0000 7f0063400700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f00480a3d50 0x7f00480a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0064076500 tx=0x7f004c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:18.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.584+0000 7f0063e00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] conn(0x7f00480ad990 0x7f00480afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:18.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.584+0000 7f0063e00700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] conn(0x7f00480ad990 0x7f00480afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:18.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.585+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== osd.10 v2:172.21.15.70:6824/3803988922 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f005401ee50 con 0x7f00480ad990 2024-04-25T18:44:18.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.603+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 --> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0064063510 con 0x7f00480ad990 2024-04-25T18:44:18.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.604+0000 7f0060c00700 1 -- 172.21.15.63:0/4261001772 <== osd.10 v2:172.21.15.70:6824/3803988922 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0064063510 con 0x7f00480ad990 2024-04-25T18:44:18.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.604+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] conn(0x7f00480ad990 msgr2=0x7f00480afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.604+0000 7f0069200700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] conn(0x7f00480ad990 0x7f00480afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f00480a3d50 msgr2=0x7f00480a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f00480a3d50 0x7f00480a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0064076500 tx=0x7f004c009260 comp rx=0 tx=0).stop 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 msgr2=0x7f0064074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 --2- 172.21.15.63:0/4261001772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0064102950 0x7f0064074d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f005400e9c0 tx=0x7f005400ecd0 comp rx=0 tx=0).stop 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0063e00700 1 -- 172.21.15.63:0/4261001772 reap_dead start 2024-04-25T18:44:18.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 shutdown_connections 2024-04-25T18:44:18.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 >> 172.21.15.63:0/4261001772 conn(0x7f00640fe420 msgr2=0x7f00640feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:18.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 shutdown_connections 2024-04-25T18:44:18.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:18.605+0000 7f0069200700 1 -- 172.21.15.63:0/4261001772 wait complete. 2024-04-25T18:44:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:18 smithi070 ceph-mon[32820]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:18.717 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:18 smithi063 ceph-mon[28006]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:18.778 INFO:teuthology.orchestra.run.smithi063.stdout:270582939706 2024-04-25T18:44:18.778 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T18:44:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:18 smithi114 ceph-mon[32988]: pgmap v398: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:20.029 INFO:teuthology.orchestra.run.smithi063.stdout:115964117076 2024-04-25T18:44:20.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T18:44:20.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.067+0000 7fe19c000700 1 -- 172.21.15.63:0/2054479496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194103450 msgr2=0x7fe19410f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.067+0000 7fe19c000700 1 --2- 172.21.15.63:0/2054479496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194103450 0x7fe19410f8e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00b0a0 tx=0x7fe18c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 -- 172.21.15.63:0/2054479496 shutdown_connections 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 --2- 172.21.15.63:0/2054479496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194108a80 0x7fe194108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 --2- 172.21.15.63:0/2054479496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194103450 0x7fe19410f8e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 --2- 172.21.15.63:0/2054479496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194102a90 0x7fe194102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 -- 172.21.15.63:0/2054479496 >> 172.21.15.63:0/2054479496 conn(0x7fe194079bf0 msgr2=0x7fe19407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 -- 172.21.15.63:0/2054479496 shutdown_connections 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.068+0000 7fe19c000700 1 -- 172.21.15.63:0/2054479496 wait complete. 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.069+0000 7fe19c000700 1 Processor -- start 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.069+0000 7fe19c000700 1 -- start start 2024-04-25T18:44:20.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.069+0000 7fe19c000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194102a90 0x7fe194074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19c000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 0x7fe194075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19c000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194108a80 0x7fe1940757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19c000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112c10 con 0x7fe194108a80 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19c000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112a90 con 0x7fe194103450 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19c000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe194112d90 con 0x7fe194102a90 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 0x7fe194075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 0x7fe194075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35100/0 (socket says 172.21.15.63:35100) 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe199600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194102a90 0x7fe194074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 -- 172.21.15.63:0/2162427644 learned_addr learned my addr 172.21.15.63:0/2162427644 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe19a000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194108a80 0x7fe1940757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 -- 172.21.15.63:0/2162427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194102a90 msgr2=0x7fe194074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe194102a90 0x7fe194074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 -- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194108a80 msgr2=0x7fe1940757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe194108a80 0x7fe1940757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.070+0000 7fe198c00700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1941a75b0 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe198c00700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 0x7fe194075280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9a0 tx=0x7fe18000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000ca10 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe18000cbb0 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1941a78a0 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180018680 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.071+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1941a7e30 con 0x7fe194103450 2024-04-25T18:44:20.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.072+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe194063150 con 0x7fe194103450 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.075+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe180010430 con 0x7fe194103450 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe192000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe17c0a3d50 0x7fe17c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fe1800d0820 con 0x7fe194103450 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe199600700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe17c0a3d50 0x7fe17c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe192000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] conn(0x7fe17c0ad990 0x7fe17c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe180024e70 con 0x7fe17c0ad990 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.076+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7fe1800d0bd0 con 0x7fe194103450 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.077+0000 7fe19a000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] conn(0x7fe17c0ad990 0x7fe17c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.077+0000 7fe199600700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe17c0a3d50 0x7fe17c0a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe188007b60 tx=0x7fe1880073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.077+0000 7fe19a000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] conn(0x7fe17c0ad990 0x7fe17c0afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.078+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== osd.4 v2:172.21.15.70:6808/2157462947 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe180024e70 con 0x7fe17c0ad990 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.097+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 --> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe194063550 con 0x7fe17c0ad990 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.098+0000 7fe192000700 1 -- 172.21.15.63:0/2162427644 <== osd.4 v2:172.21.15.70:6808/2157462947 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe194063550 con 0x7fe17c0ad990 2024-04-25T18:44:20.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.098+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] conn(0x7fe17c0ad990 msgr2=0x7fe17c0afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.098+0000 7fe19c000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] conn(0x7fe17c0ad990 0x7fe17c0afe30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe17c0a3d50 msgr2=0x7fe17c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe17c0a3d50 0x7fe17c0a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe188007b60 tx=0x7fe1880073d0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 msgr2=0x7fe194075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 --2- 172.21.15.63:0/2162427644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe194103450 0x7fe194075280 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9a0 tx=0x7fe18000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19a000700 1 -- 172.21.15.63:0/2162427644 reap_dead start 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 shutdown_connections 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 >> 172.21.15.63:0/2162427644 conn(0x7fe194079bf0 msgr2=0x7fe194101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 shutdown_connections 2024-04-25T18:44:20.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:20.099+0000 7fe19c000700 1 -- 172.21.15.63:0/2162427644 wait complete. 2024-04-25T18:44:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:20 smithi070 ceph-mon[32820]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:20 smithi114 ceph-mon[32988]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:20.859 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:20 smithi063 ceph-mon[28006]: pgmap v399: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:21.051 INFO:teuthology.orchestra.run.smithi063.stdout:304942678067 2024-04-25T18:44:21.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T18:44:21.473 INFO:teuthology.orchestra.run.smithi063.stdout:150323855440 2024-04-25T18:44:21.473 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T18:44:22.055 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.066 INFO:teuthology.orchestra.run.smithi063.stdout:326417514542 2024-04-25T18:44:22.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T18:44:22.068 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.069 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.074 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.083 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.530 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.531 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.531 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.599 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:22 smithi063 ceph-mon[28006]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:22 smithi070 ceph-mon[32820]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:22.728 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:22 smithi114 ceph-mon[32988]: pgmap v400: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:23.262 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:23.263 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:24.477 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:24 smithi063 ceph-mon[28006]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T18:44:24.478 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:24 smithi063 ceph-mon[28006]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-25T18:44:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:24 smithi070 ceph-mon[32820]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T18:44:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:24 smithi070 ceph-mon[32820]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-25T18:44:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:24 smithi114 ceph-mon[32988]: pgmap v401: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-25T18:44:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:24 smithi114 ceph-mon[32988]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-25T18:44:25.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:25 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:25 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:25.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:25 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:44:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:26 smithi070 ceph-mon[32820]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:44:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:26 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:26 smithi063 ceph-mon[28006]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:44:26.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:26.983 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:26 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:26 smithi114 ceph-mon[32988]: pgmap v402: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:44:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:26 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.981 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:27.982 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:27 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:27 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:28.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:27 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:29.010 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:28 smithi063 ceph-mon[28006]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T18:44:29.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:28 smithi114 ceph-mon[32988]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T18:44:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:28 smithi070 ceph-mon[32820]: pgmap v403: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 85 B/s wr, 4 op/s 2024-04-25T18:44:30.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.044+0000 7f1513c00700 1 -- 172.21.15.63:0/2255141981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c103450 msgr2=0x7f150c10f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:30.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.044+0000 7f1513c00700 1 --2- 172.21.15.63:0/2255141981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c103450 0x7f150c10f8b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1504009f90 tx=0x7f150400b330 comp rx=0 tx=0).stop 2024-04-25T18:44:30.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 -- 172.21.15.63:0/2255141981 shutdown_connections 2024-04-25T18:44:30.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 --2- 172.21.15.63:0/2255141981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c103450 0x7f150c10f8b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 --2- 172.21.15.63:0/2255141981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c108a50 0x7f150c108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 --2- 172.21.15.63:0/2255141981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 -- 172.21.15.63:0/2255141981 >> 172.21.15.63:0/2255141981 conn(0x7f150c079ba0 msgr2=0x7f150c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 -- 172.21.15.63:0/2255141981 shutdown_connections 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.045+0000 7f1513c00700 1 -- 172.21.15.63:0/2255141981 wait complete. 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.046+0000 7f1513c00700 1 Processor -- start 2024-04-25T18:44:30.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 -- start start 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 0x7f150c19b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f150c112c50 con 0x7f150c103450 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f150c112ad0 con 0x7f150c102a90 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1513c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f150c112dd0 con 0x7f150c108a50 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f150be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 0x7f150c19b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1511c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.047+0000 7f1511200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:30.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48970/0 (socket says 172.21.15.63:48970) 2024-04-25T18:44:30.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f150be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 0x7f150c19b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37216/0 (socket says 172.21.15.63:37216) 2024-04-25T18:44:30.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c19b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47770/0 (socket says 172.21.15.63:47770) 2024-04-25T18:44:30.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 -- 172.21.15.63:0/1894584237 learned_addr learned my addr 172.21.15.63:0/1894584237 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:30.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 -- 172.21.15.63:0/1894584237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 msgr2=0x7f150c19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 -- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 msgr2=0x7f150c19b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 0x7f150c19b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 -- 172.21.15.63:0/1894584237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c1a02b0 con 0x7f150c108a50 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.048+0000 7f1511c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1504009c10 tx=0x7f150400cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.049+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504021070 con 0x7f150c108a50 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.049+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c1a0540 con 0x7f150c108a50 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.049+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1504007ac0 con 0x7f150c108a50 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.049+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504005040 con 0x7f150c108a50 2024-04-25T18:44:30.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.049+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c1a7e20 con 0x7f150c108a50 2024-04-25T18:44:30.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.050+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c069730 con 0x7f150c108a50 2024-04-25T18:44:30.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.054+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1504024070 con 0x7f150c108a50 2024-04-25T18:44:30.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.055+0000 7f1509600700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 0x7f14f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:30.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.056+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f15040d1240 con 0x7f150c108a50 2024-04-25T18:44:30.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.056+0000 7f1511200700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 0x7f14f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:30.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.056+0000 7f1511200700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 0x7f14f40a6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1500009be0 tx=0x7f1500009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:30.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.060+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f150409dc80 con 0x7f150c108a50 2024-04-25T18:44:30.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.295+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f150c02df80 con 0x7f150c108a50 2024-04-25T18:44:30.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.297+0000 7f1509600700 1 -- 172.21.15.63:0/1894584237 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f150409d580 con 0x7f150c108a50 2024-04-25T18:44:30.299 INFO:teuthology.orchestra.run.smithi063.stdout:115964117079 2024-04-25T18:44:30.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 msgr2=0x7f14f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 0x7f14f40a6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1500009be0 tx=0x7f1500009260 comp rx=0 tx=0).stop 2024-04-25T18:44:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 msgr2=0x7f150c19fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1504009c10 tx=0x7f150400cca0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 shutdown_connections 2024-04-25T18:44:30.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f14f40a3cc0 0x7f14f40a6180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f150c108a50 0x7f150c19fb70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150c102a90 0x7f150c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 --2- 172.21.15.63:0/1894584237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f150c103450 0x7f150c19b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 >> 172.21.15.63:0/1894584237 conn(0x7f150c079ba0 msgr2=0x7f150c10dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 shutdown_connections 2024-04-25T18:44:30.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:30.300+0000 7f1513c00700 1 -- 172.21.15.63:0/1894584237 wait complete. 2024-04-25T18:44:31.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:30 smithi114 ceph-mon[32988]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:44:31.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:30 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1894584237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T18:44:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:30 smithi063 ceph-mon[28006]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:44:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:30 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1894584237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T18:44:31.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:30 smithi070 ceph-mon[32820]: pgmap v404: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:44:31.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:30 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1894584237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T18:44:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:32 smithi114 ceph-mon[32988]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:44:33.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:32 smithi063 ceph-mon[28006]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:44:33.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:32 smithi070 ceph-mon[32820]: pgmap v405: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.741+0000 7fee25e00700 1 -- 172.21.15.63:0/2328863433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 msgr2=0x7fee20115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.741+0000 7fee25e00700 1 --2- 172.21.15.63:0/2328863433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee20115f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fee1800b0d0 tx=0x7fee1800b3e0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 -- 172.21.15.63:0/2328863433 shutdown_connections 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 --2- 172.21.15.63:0/2328863433 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee201092d0 0x7fee201096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 --2- 172.21.15.63:0/2328863433 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee20115f80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 --2- 172.21.15.63:0/2328863433 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee20109c80 0x7fee2010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 -- 172.21.15.63:0/2328863433 >> 172.21.15.63:0/2328863433 conn(0x7fee20076210 msgr2=0x7fee20107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:33.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 -- 172.21.15.63:0/2328863433 shutdown_connections 2024-04-25T18:44:33.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.742+0000 7fee25e00700 1 -- 172.21.15.63:0/2328863433 wait complete. 2024-04-25T18:44:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 Processor -- start 2024-04-25T18:44:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 -- start start 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 0x7fee2019f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee20109c80 0x7fee2019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee201192a0 con 0x7fee2010e3b0 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee20119120 con 0x7fee201092d0 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.743+0000 7fee25e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee20119420 con 0x7fee20109c80 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee20109c80 0x7fee2019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 0x7fee2019f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37226/0 (socket says 172.21.15.63:37226) 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 -- 172.21.15.63:0/132546278 learned_addr learned my addr 172.21.15.63:0/132546278 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 0x7fee2019f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47792/0 (socket says 172.21.15.63:47792) 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 -- 172.21.15.63:0/132546278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee20109c80 msgr2=0x7fee2019fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee20109c80 0x7fee2019fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 -- 172.21.15.63:0/132546278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 msgr2=0x7fee2019f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 0x7fee2019f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.744+0000 7fee1f400700 1 -- 172.21.15.63:0/132546278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee201a46e0 con 0x7fee2010e3b0 2024-04-25T18:44:33.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee1f400700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee180079a0 tx=0x7fee180079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1801f070 con 0x7fee2010e3b0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee18003c00 con 0x7fee2010e3b0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1801b690 con 0x7fee2010e3b0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee201a4970 con 0x7fee2010e3b0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.745+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee201ac250 con 0x7fee2010e3b0 2024-04-25T18:44:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.746+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee20069730 con 0x7fee2010e3b0 2024-04-25T18:44:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.748+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fee18022070 con 0x7fee2010e3b0 2024-04-25T18:44:33.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.750+0000 7fee24a00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 0x7fee040a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:33.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.750+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fee1800f020 con 0x7fee2010e3b0 2024-04-25T18:44:33.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.750+0000 7fee1ea00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 0x7fee040a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:33.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.751+0000 7fee1ea00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 0x7fee040a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee10007be0 tx=0x7fee100073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:33.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.755+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee18029020 con 0x7fee2010e3b0 2024-04-25T18:44:33.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.994+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fee2002df80 con 0x7fee2010e3b0 2024-04-25T18:44:33.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.994+0000 7fee24a00700 1 -- 172.21.15.63:0/132546278 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fee1809dea0 con 0x7fee2010e3b0 2024-04-25T18:44:33.996 INFO:teuthology.orchestra.run.smithi063.stdout:231928234050 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.997+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 msgr2=0x7fee040a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 0x7fee040a6180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fee10007be0 tx=0x7fee100073d0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 msgr2=0x7fee201a3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee180079a0 tx=0x7fee180079d0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 shutdown_connections 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee040a3cc0 0x7fee040a6180 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2010e3b0 0x7fee201a3fa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee201092d0 0x7fee2019f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 --2- 172.21.15.63:0/132546278 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee20109c80 0x7fee2019fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:33.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 >> 172.21.15.63:0/132546278 conn(0x7fee20076210 msgr2=0x7fee20076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:34.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 shutdown_connections 2024-04-25T18:44:34.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:33.998+0000 7fee25e00700 1 -- 172.21.15.63:0/132546278 wait complete. 2024-04-25T18:44:34.392 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117076 got 115964117079 for osd.3 2024-04-25T18:44:34.393 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:34.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:34 smithi063 ceph-mon[28006]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:34.860 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:34 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/132546278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T18:44:35.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:34 smithi114 ceph-mon[32988]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:35.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:34 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/132546278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T18:44:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:34 smithi070 ceph-mon[32820]: pgmap v406: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:35.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:34 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/132546278' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T18:44:37.049 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:36 smithi063 ceph-mon[28006]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:37.049 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:36 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3401754047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T18:44:37.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:36 smithi114 ceph-mon[32988]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:36 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3401754047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T18:44:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:36 smithi070 ceph-mon[32820]: pgmap v407: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-04-25T18:44:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:36 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3401754047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T18:44:37.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 -- 172.21.15.63:0/3548362336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e0074c70 msgr2=0x7f74e0075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 --2- 172.21.15.63:0/3548362336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e0074c70 0x7f74e0075050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009990 tx=0x7f74d4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 -- 172.21.15.63:0/3548362336 shutdown_connections 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 --2- 172.21.15.63:0/3548362336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e010f440 0x7f74e0111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 --2- 172.21.15.63:0/3548362336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e0074c70 0x7f74e0075050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 --2- 172.21.15.63:0/3548362336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0075590 0x7f74e010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.569+0000 7f74e7a00700 1 -- 172.21.15.63:0/3548362336 >> 172.21.15.63:0/3548362336 conn(0x7f74e0079ba0 msgr2=0x7f74e0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.570+0000 7f74e7a00700 1 -- 172.21.15.63:0/3548362336 shutdown_connections 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.570+0000 7f74e7a00700 1 -- 172.21.15.63:0/3548362336 wait complete. 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.571+0000 7f74e7a00700 1 Processor -- start 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.571+0000 7f74e7a00700 1 -- start start 2024-04-25T18:44:37.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e0075590 0x7f74e019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 0x7f74e019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0114e70 con 0x7f74e010f440 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0114cf0 con 0x7f74e0075590 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e7a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0114ff0 con 0x7f74e0074c70 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e5000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e5000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37024/0 (socket says 172.21.15.63:37024) 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74e5000700 1 -- 172.21.15.63:0/1388971185 learned_addr learned my addr 172.21.15.63:0/1388971185 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.572+0000 7f74dfe00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e0075590 0x7f74e019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 0x7f74e019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 -- 172.21.15.63:0/1388971185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e0075590 msgr2=0x7f74e019b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:37.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e0075590 0x7f74e019b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 -- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 msgr2=0x7f74e019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 0x7f74e019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 -- 172.21.15.63:0/1388971185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e01a0410 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 0x7f74e019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.573+0000 7f74e5000700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f74d4005910 tx=0x7f74d400b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.574+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4027070 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.574+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d4022470 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.574+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e01a06a0 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.574+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4017650 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.574+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e01a7f80 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.575+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e0076360 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.577+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f74d402a070 con 0x7f74e0074c70 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.579+0000 7f74dd600700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 0x7f74c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:37.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.579+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f74d40d11d0 con 0x7f74e0074c70 2024-04-25T18:44:37.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.579+0000 7f74dfe00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 0x7f74c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:37.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.579+0000 7f74dfe00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 0x7f74c80a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f74e0101dd0 tx=0x7f74cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:37.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.583+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74d4013070 con 0x7f74e0074c70 2024-04-25T18:44:37.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.831+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f74e002df80 con 0x7f74e0074c70 2024-04-25T18:44:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.832+0000 7f74dd600700 1 -- 172.21.15.63:0/1388971185 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f74d409dbb0 con 0x7f74e0074c70 2024-04-25T18:44:37.833 INFO:teuthology.orchestra.run.smithi063.stdout:304942678071 2024-04-25T18:44:37.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.834+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 msgr2=0x7f74c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.834+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 0x7f74c80a6100 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f74e0101dd0 tx=0x7f74cc009260 comp rx=0 tx=0).stop 2024-04-25T18:44:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.834+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 msgr2=0x7f74e019b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.834+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f74d4005910 tx=0x7f74d400b670 comp rx=0 tx=0).stop 2024-04-25T18:44:37.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 shutdown_connections 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f74c80a3c40 0x7f74c80a6100 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74e0074c70 0x7f74e019b400 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e010f440 0x7f74e019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 --2- 172.21.15.63:0/1388971185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74e0075590 0x7f74e019b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 >> 172.21.15.63:0/1388971185 conn(0x7f74e0079ba0 msgr2=0x7f74e0101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 shutdown_connections 2024-04-25T18:44:37.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:37.835+0000 7f74e7a00700 1 -- 172.21.15.63:0/1388971185 wait complete. 2024-04-25T18:44:39.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:38 smithi063 ceph-mon[28006]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:44:39.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:38 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1388971185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T18:44:39.090 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:38 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/282059665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 -- 172.21.15.63:0/4114517727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 msgr2=0x7fb02410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 --2- 172.21.15.63:0/4114517727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02410ef00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb010009990 tx=0x7fb010009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 -- 172.21.15.63:0/4114517727 shutdown_connections 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 --2- 172.21.15.63:0/4114517727 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb02410f440 0x7fb024111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 --2- 172.21.15.63:0/4114517727 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb024074c70 0x7fb024075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 --2- 172.21.15.63:0/4114517727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02410ef00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 -- 172.21.15.63:0/4114517727 >> 172.21.15.63:0/4114517727 conn(0x7fb024079bf0 msgr2=0x7fb02407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 -- 172.21.15.63:0/4114517727 shutdown_connections 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.161+0000 7fb02b400700 1 -- 172.21.15.63:0/4114517727 wait complete. 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.163+0000 7fb02b400700 1 Processor -- start 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.163+0000 7fb02b400700 1 -- start start 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb024074c70 0x7fb02419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 0x7fb02419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb024114f10 con 0x7fb024075590 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb024114d90 con 0x7fb02410f440 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb02b400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb024115090 con 0x7fb024074c70 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb023e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb023e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45722/0 (socket says 172.21.15.63:45722) 2024-04-25T18:44:39.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb028a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb024074c70 0x7fb02419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb023e00700 1 -- 172.21.15.63:0/2800277912 learned_addr learned my addr 172.21.15.63:0/2800277912 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.164+0000 7fb029400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 0x7fb02419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 -- 172.21.15.63:0/2800277912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb024074c70 msgr2=0x7fb02419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb024074c70 0x7fb02419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 -- 172.21.15.63:0/2800277912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 msgr2=0x7fb02419fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 0x7fb02419fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 -- 172.21.15.63:0/2800277912 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0241a03d0 con 0x7fb024075590 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb029400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 0x7fb02419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb023e00700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb010000c00 tx=0x7fb0100057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010027070 con 0x7fb024075590 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0241a0660 con 0x7fb024075590 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb010022470 con 0x7fb024075590 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.165+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010017650 con 0x7fb024075590 2024-04-25T18:44:39.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.166+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0241a7f40 con 0x7fb024075590 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.167+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb024069730 con 0x7fb024075590 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.169+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb01002a070 con 0x7fb024075590 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.170+0000 7fb021600700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 0x7fb00c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.170+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fb0100d10c0 con 0x7fb024075590 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.170+0000 7fb028a00700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 0x7fb00c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.171+0000 7fb028a00700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 0x7fb00c0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb018009c60 tx=0x7fb018009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:39.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.175+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb010013070 con 0x7fb024075590 2024-04-25T18:44:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:38 smithi070 ceph-mon[32820]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:44:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:38 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1388971185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T18:44:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:38 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/282059665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T18:44:39.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:38 smithi114 ceph-mon[32988]: pgmap v408: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:44:39.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:38 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1388971185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T18:44:39.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:38 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/282059665' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T18:44:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.415+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb02402df80 con 0x7fb024075590 2024-04-25T18:44:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.416+0000 7fb021600700 1 -- 172.21.15.63:0/2800277912 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb01009db00 con 0x7fb024075590 2024-04-25T18:44:39.417 INFO:teuthology.orchestra.run.smithi063.stdout:150323855443 2024-04-25T18:44:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 msgr2=0x7fb00c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 0x7fb00c0a6100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb018009c60 tx=0x7fb018009260 comp rx=0 tx=0).stop 2024-04-25T18:44:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 msgr2=0x7fb02419b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb010000c00 tx=0x7fb0100057c0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 shutdown_connections 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fb00c0a3c40 0x7fb00c0a6100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb024074c70 0x7fb02419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb02410f440 0x7fb02419fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 --2- 172.21.15.63:0/2800277912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb024075590 0x7fb02419b900 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 >> 172.21.15.63:0/2800277912 conn(0x7fb024079bf0 msgr2=0x7fb0240ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 shutdown_connections 2024-04-25T18:44:39.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:39.419+0000 7fb02b400700 1 -- 172.21.15.63:0/2800277912 wait complete. 2024-04-25T18:44:39.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234043 got 231928234050 for osd.7 2024-04-25T18:44:39.620 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:39.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:39 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2800277912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T18:44:40.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:39 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2800277912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T18:44:40.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:39 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2800277912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T18:44:40.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 -- 172.21.15.63:0/376275242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 msgr2=0x7f829810f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:40.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 --2- 172.21.15.63:0/376275242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f829810f8e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f829000b0a0 tx=0x7f829000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 -- 172.21.15.63:0/376275242 shutdown_connections 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 --2- 172.21.15.63:0/376275242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f829810f8e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 --2- 172.21.15.63:0/376275242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f8298108df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 --2- 172.21.15.63:0/376275242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 -- 172.21.15.63:0/376275242 >> 172.21.15.63:0/376275242 conn(0x7f8298079c00 msgr2=0x7f829807a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.312+0000 7f829ce00700 1 -- 172.21.15.63:0/376275242 shutdown_connections 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.313+0000 7f829ce00700 1 -- 172.21.15.63:0/376275242 wait complete. 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 Processor -- start 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 -- start start 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f82981995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:40.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f829819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8298112bf0 con 0x7f82981034c0 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8298112a70 con 0x7f8298108a10 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.314+0000 7f829ce00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8298112d70 con 0x7f8298102b00 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f82981995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37036/0 (socket says 172.21.15.63:37036) 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 -- 172.21.15.63:0/282059665 learned_addr learned my addr 172.21.15.63:0/282059665 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8296800700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f829819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 -- 172.21.15.63:0/282059665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 msgr2=0x7f829819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f829819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 -- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 msgr2=0x7f82981995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f82981995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 -- 172.21.15.63:0/282059665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829819e020 con 0x7f8298102b00 2024-04-25T18:44:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8296800700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f829819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.315+0000 7f8295e00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f828800eb30 tx=0x7f828800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.316+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828800cbd0 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.316+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829819e310 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.316+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82880044d0 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.316+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8288003e30 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.316+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82981a5b90 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.317+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8298069730 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.319+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8288004000 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.320+0000 7f828ea00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 0x7f827c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.321+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f82880d0780 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.321+0000 7f8295400700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 0x7f827c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.321+0000 7f8295400700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 0x7f827c0a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f829807a170 tx=0x7f8284009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.325+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f82880042f0 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.566+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f829802df80 con 0x7f8298102b00 2024-04-25T18:44:40.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.567+0000 7f828ea00700 1 -- 172.21.15.63:0/282059665 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f828809d0d0 con 0x7f8298102b00 2024-04-25T18:44:40.873 INFO:teuthology.orchestra.run.smithi063.stdout:55834574949 2024-04-25T18:44:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 msgr2=0x7f827c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 0x7f827c0a6100 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f829807a170 tx=0x7f8284009260 comp rx=0 tx=0).stop 2024-04-25T18:44:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 msgr2=0x7f8298199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f828800eb30 tx=0x7f828800ee40 comp rx=0 tx=0).stop 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 shutdown_connections 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f827c0a3c40 0x7f827c0a6100 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8298102b00 0x7f8298199060 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8298108a10 0x7f829819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 --2- 172.21.15.63:0/282059665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82981034c0 0x7f82981995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 >> 172.21.15.63:0/282059665 conn(0x7f8298079c00 msgr2=0x7f8298100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 shutdown_connections 2024-04-25T18:44:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:38.570+0000 7f829ce00700 1 -- 172.21.15.63:0/282059665 wait complete. 2024-04-25T18:44:41.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:40 smithi063 ceph-mon[28006]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-25T18:44:41.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:40 smithi070 ceph-mon[32820]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-25T18:44:41.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:40 smithi114 ceph-mon[32988]: pgmap v409: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-25T18:44:41.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 -- 172.21.15.63:0/291472538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804104ec0 msgr2=0x7f88041052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 --2- 172.21.15.63:0/291472538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804104ec0 0x7f88041052a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f87f40099c0 tx=0x7f87f4009cd0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 -- 172.21.15.63:0/291472538 shutdown_connections 2024-04-25T18:44:41.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 --2- 172.21.15.63:0/291472538 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804109fa0 0x7f8804111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 --2- 172.21.15.63:0/291472538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804104ec0 0x7f88041052a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 --2- 172.21.15.63:0/291472538 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 0x7f88041098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.509+0000 7f8809200700 1 -- 172.21.15.63:0/291472538 >> 172.21.15.63:0/291472538 conn(0x7f8804100750 msgr2=0x7f8804102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:41.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.510+0000 7f8809200700 1 -- 172.21.15.63:0/291472538 shutdown_connections 2024-04-25T18:44:41.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.510+0000 7f8809200700 1 -- 172.21.15.63:0/291472538 wait complete. 2024-04-25T18:44:41.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 Processor -- start 2024-04-25T18:44:41.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 -- start start 2024-04-25T18:44:41.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 0x7f880419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 0x7f880419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8804114ea0 con 0x7f8804109fa0 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8804114d20 con 0x7f8804104ec0 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.511+0000 7f8809200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8804115020 con 0x7f8804105870 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8802800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 0x7f880419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57458/0 (socket says 172.21.15.63:57458) 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 -- 172.21.15.63:0/351381750 learned_addr learned my addr 172.21.15.63:0/351381750 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:41.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f87fbe00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 0x7f880419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 -- 172.21.15.63:0/351381750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 msgr2=0x7f880419b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 0x7f880419b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 -- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 msgr2=0x7f880419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 0x7f880419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8801e00700 1 -- 172.21.15.63:0/351381750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88041a0380 con 0x7f8804104ec0 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.512+0000 7f8802800700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 0x7f880419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:41.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.513+0000 7f8801e00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006010 tx=0x7f87f4005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.513+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f4026070 con 0x7f8804104ec0 2024-04-25T18:44:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.513+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87f4005680 con 0x7f8804104ec0 2024-04-25T18:44:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.513+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88041a0610 con 0x7f8804104ec0 2024-04-25T18:44:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.514+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87f400fb20 con 0x7f8804104ec0 2024-04-25T18:44:41.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.514+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88041a7ef0 con 0x7f8804104ec0 2024-04-25T18:44:41.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.515+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f880419c6d0 con 0x7f8804104ec0 2024-04-25T18:44:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.517+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f87f4029070 con 0x7f8804104ec0 2024-04-25T18:44:41.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.518+0000 7f87faa00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 0x7f87e40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.518+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f87f40d12a0 con 0x7f8804104ec0 2024-04-25T18:44:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.519+0000 7f87fbe00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 0x7f87e40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.519+0000 7f87fbe00700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 0x7f87e40a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f880410b550 tx=0x7f87ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.521+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87f4030020 con 0x7f8804104ec0 2024-04-25T18:44:41.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.753+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8804069710 con 0x7f8804104ec0 2024-04-25T18:44:41.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.753+0000 7f87faa00700 1 -- 172.21.15.63:0/351381750 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f87f4013070 con 0x7f8804104ec0 2024-04-25T18:44:41.755 INFO:teuthology.orchestra.run.smithi063.stdout:34359738477 2024-04-25T18:44:41.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.756+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 msgr2=0x7f87e40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.756+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 0x7f87e40a6180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f880410b550 tx=0x7f87ec009260 comp rx=0 tx=0).stop 2024-04-25T18:44:41.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.756+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 msgr2=0x7f880419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.756+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006010 tx=0x7f87f4005280 comp rx=0 tx=0).stop 2024-04-25T18:44:41.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 shutdown_connections 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f87e40a3cc0 0x7f87e40a6180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8804109fa0 0x7f880419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8804104ec0 0x7f880419b370 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 --2- 172.21.15.63:0/351381750 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8804105870 0x7f880419b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 >> 172.21.15.63:0/351381750 conn(0x7f8804100750 msgr2=0x7f8804102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 shutdown_connections 2024-04-25T18:44:41.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.757+0000 7f8809200700 1 -- 172.21.15.63:0/351381750 wait complete. 2024-04-25T18:44:41.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.954+0000 7fee65400700 1 -- 172.21.15.63:0/1469309669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee6010e3b0 msgr2=0x7fee60115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.954+0000 7fee65400700 1 --2- 172.21.15.63:0/1469309669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee6010e3b0 0x7fee60115f80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee58009fc0 tx=0x7fee5800b330 comp rx=0 tx=0).stop 2024-04-25T18:44:41.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 -- 172.21.15.63:0/1469309669 shutdown_connections 2024-04-25T18:44:41.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 --2- 172.21.15.63:0/1469309669 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee6010e3b0 0x7fee60115f80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 --2- 172.21.15.63:0/1469309669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee601092d0 0x7fee601096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 --2- 172.21.15.63:0/1469309669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee60109c80 0x7fee6010dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 -- 172.21.15.63:0/1469309669 >> 172.21.15.63:0/1469309669 conn(0x7fee60076210 msgr2=0x7fee60107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 -- 172.21.15.63:0/1469309669 shutdown_connections 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.955+0000 7fee65400700 1 -- 172.21.15.63:0/1469309669 wait complete. 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 Processor -- start 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 -- start start 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee601092d0 0x7fee6019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60109c80 0x7fee6019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee601192a0 con 0x7fee6010e3b0 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee60119120 con 0x7fee601092d0 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.956+0000 7fee65400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee60119420 con 0x7fee60109c80 2024-04-25T18:44:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45736/0 (socket says 172.21.15.63:45736) 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 -- 172.21.15.63:0/3397762940 learned_addr learned my addr 172.21.15.63:0/3397762940 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5de00700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60109c80 0x7fee6019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5e800700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee601092d0 0x7fee6019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 -- 172.21.15.63:0/3397762940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60109c80 msgr2=0x7fee6019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60109c80 0x7fee6019fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 -- 172.21.15.63:0/3397762940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee601092d0 msgr2=0x7fee6019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee601092d0 0x7fee6019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 -- 172.21.15.63:0/3397762940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee601a46c0 con 0x7fee6010e3b0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.957+0000 7fee5f200700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee58009790 tx=0x7fee5800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.958+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58021070 con 0x7fee6010e3b0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.958+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee5800ce60 con 0x7fee6010e3b0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.958+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee601a4950 con 0x7fee6010e3b0 2024-04-25T18:44:41.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.958+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58005040 con 0x7fee6010e3b0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.958+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee601ac230 con 0x7fee6010e3b0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.959+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee60069730 con 0x7fee6010e3b0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.963+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fee58024070 con 0x7fee6010e3b0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.964+0000 7fee57400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 0x7fee440a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.964+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fee5800f020 con 0x7fee6010e3b0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.964+0000 7fee5e800700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 0x7fee440a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.965+0000 7fee5e800700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 0x7fee440a6180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee50009750 tx=0x7fee50009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:41.921 INFO:teuthology.orchestra.run.smithi063.stdout:360777252909 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:40.967+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee5802b020 con 0x7fee6010e3b0 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.208+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fee6002df80 con 0x7fee6010e3b0 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.208+0000 7fee57400700 1 -- 172.21.15.63:0/3397762940 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fee5809e5d0 con 0x7fee6010e3b0 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 msgr2=0x7fee440a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 0x7fee440a6180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fee50009750 tx=0x7fee50009310 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 msgr2=0x7fee601a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee58009790 tx=0x7fee5800cc30 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 shutdown_connections 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fee440a3cc0 0x7fee440a6180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee6010e3b0 0x7fee601a3f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee601092d0 0x7fee6019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 --2- 172.21.15.63:0/3397762940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee60109c80 0x7fee6019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:41.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.211+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 >> 172.21.15.63:0/3397762940 conn(0x7fee60076210 msgr2=0x7fee600767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:41.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.212+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 shutdown_connections 2024-04-25T18:44:41.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:41.212+0000 7fee65400700 1 -- 172.21.15.63:0/3397762940 wait complete. 2024-04-25T18:44:42.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:41 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3397762940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T18:44:42.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:41 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/351381750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T18:44:42.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:41 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3397762940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T18:44:42.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:41 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/351381750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T18:44:42.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:41 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3397762940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T18:44:42.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:41 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/351381750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T18:44:42.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574943 got 55834574949 for osd.1 2024-04-25T18:44:42.474 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:42.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855440 got 150323855443 for osd.4 2024-04-25T18:44:42.666 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:42.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 -- 172.21.15.63:0/633150735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0105870 msgr2=0x7f27b01098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 --2- 172.21.15.63:0/633150735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0105870 0x7f27b01098c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f27980099c0 tx=0x7f2798009cd0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 -- 172.21.15.63:0/633150735 shutdown_connections 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 --2- 172.21.15.63:0/633150735 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0109fa0 0x7f27b0111b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 --2- 172.21.15.63:0/633150735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 0x7f27b01052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 --2- 172.21.15.63:0/633150735 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0105870 0x7f27b01098c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 -- 172.21.15.63:0/633150735 >> 172.21.15.63:0/633150735 conn(0x7f27b0100750 msgr2=0x7f27b0102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 -- 172.21.15.63:0/633150735 shutdown_connections 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.081+0000 7f27b5a00700 1 -- 172.21.15.63:0/633150735 wait complete. 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.082+0000 7f27b5a00700 1 Processor -- start 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.082+0000 7f27b5a00700 1 -- start start 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 0x7f27b019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0105870 0x7f27b019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:42.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0114e30 con 0x7f27b0109fa0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0114cb0 con 0x7f27b0104ec0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27b5a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f27b0114fb0 con 0x7f27b0105870 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27af400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27af400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37250/0 (socket says 172.21.15.63:37250) 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27af400700 1 -- 172.21.15.63:0/3401754047 learned_addr learned my addr 172.21.15.63:0/3401754047 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27aea00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 0x7f27b019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.083+0000 7f27a7e00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0105870 0x7f27b019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 -- 172.21.15.63:0/3401754047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0105870 msgr2=0x7f27b019b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0105870 0x7f27b019b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 -- 172.21.15.63:0/3401754047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 msgr2=0x7f27b019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 0x7f27b019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 -- 172.21.15.63:0/3401754047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b01a0310 con 0x7f27b0109fa0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27af400700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f27a800ef30 tx=0x7f27a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:42.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a80092e0 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b01a05a0 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27a8004bb0 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.084+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a8004280 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.085+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b01a7e80 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.086+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b0069730 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.088+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f27a801b030 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.090+0000 7f27a7400700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 0x7f27940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.090+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f27a80cfee0 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.090+0000 7f27aea00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 0x7f27940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.091+0000 7f27aea00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 0x7f27940a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f27a0005950 tx=0x7f27a000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.095+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27a8022020 con 0x7f27b0109fa0 2024-04-25T18:44:42.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.330+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f27b002e140 con 0x7f27b0109fa0 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stdout:210453397576 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.330+0000 7f27a7400700 1 -- 172.21.15.63:0/3401754047 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27a8010040 con 0x7f27b0109fa0 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 msgr2=0x7f27940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 0x7f27940a6100 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f27a0005950 tx=0x7f27a000b410 comp rx=0 tx=0).stop 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 msgr2=0x7f27b019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f27a800ef30 tx=0x7f27a800c3a0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 shutdown_connections 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f27940a3c40 0x7f27940a6100 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b0109fa0 0x7f27b019fbd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27b0104ec0 0x7f27b019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 --2- 172.21.15.63:0/3401754047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f27b0105870 0x7f27b019b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.333+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 >> 172.21.15.63:0/3401754047 conn(0x7f27b0100750 msgr2=0x7f27b0102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.334+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 shutdown_connections 2024-04-25T18:44:42.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:36.334+0000 7f27b5a00700 1 -- 172.21.15.63:0/3401754047 wait complete. 2024-04-25T18:44:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:42 smithi063 ceph-mon[28006]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:43.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:42 smithi070 ceph-mon[32820]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:43.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:42 smithi114 ceph-mon[32988]: pgmap v410: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:43.550 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:44.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397569 got 210453397576 for osd.6 2024-04-25T18:44:44.014 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:44.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 -- 172.21.15.63:0/94350861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133006a750 msgr2=0x7f133006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:44.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 --2- 172.21.15.63:0/94350861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133006a750 0x7f133006ab30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1324009990 tx=0x7f1324009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 -- 172.21.15.63:0/94350861 shutdown_connections 2024-04-25T18:44:44.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 --2- 172.21.15.63:0/94350861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133010f4f0 0x7f13301118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 --2- 172.21.15.63:0/94350861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133006a750 0x7f133006ab30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 --2- 172.21.15.63:0/94350861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006b070 0x7f133010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 -- 172.21.15.63:0/94350861 >> 172.21.15.63:0/94350861 conn(0x7f13300788f0 msgr2=0x7f1330078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 -- 172.21.15.63:0/94350861 shutdown_connections 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.691+0000 7f1337600700 1 -- 172.21.15.63:0/94350861 wait complete. 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.693+0000 7f1337600700 1 Processor -- start 2024-04-25T18:44:44.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.693+0000 7f1337600700 1 -- start start 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133006b070 0x7f133019b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 0x7f133019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114e20 con 0x7f133006b070 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114ca0 con 0x7f133010f4f0 2024-04-25T18:44:44.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1337600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1330114fa0 con 0x7f133006a750 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1334c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f132fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133006b070 0x7f133019b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1334c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37122/0 (socket says 172.21.15.63:37122) 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1334c00700 1 -- 172.21.15.63:0/4133553619 learned_addr learned my addr 172.21.15.63:0/4133553619 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.694+0000 7f1335600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 0x7f133019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 -- 172.21.15.63:0/4133553619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 msgr2=0x7f133019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 0x7f133019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 -- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133006b070 msgr2=0x7f133019b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133006b070 0x7f133019b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 -- 172.21.15.63:0/4133553619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13301a0340 con 0x7f133006a750 2024-04-25T18:44:44.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1335600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 0x7f133019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:44:44.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f1334c00700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13240057f0 tx=0x7f132400bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:44.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.695+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324026070 con 0x7f133006a750 2024-04-25T18:44:44.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.696+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13301a05d0 con 0x7f133006a750 2024-04-25T18:44:44.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.696+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1324021410 con 0x7f133006a750 2024-04-25T18:44:44.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.696+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324017650 con 0x7f133006a750 2024-04-25T18:44:44.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.696+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13301a7eb0 con 0x7f133006a750 2024-04-25T18:44:44.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.697+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1330069730 con 0x7f133006a750 2024-04-25T18:44:44.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.699+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1324029070 con 0x7f133006a750 2024-04-25T18:44:44.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.700+0000 7f132d600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 0x7f13180a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:44.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.701+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f13240d1170 con 0x7f133006a750 2024-04-25T18:44:44.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.701+0000 7f132fe00700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 0x7f13180a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:44.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.702+0000 7f132fe00700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 0x7f13180a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1330102f00 tx=0x7f131c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:44.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.705+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f132405dff0 con 0x7f133006a750 2024-04-25T18:44:44.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.956+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f133002df80 con 0x7f133006a750 2024-04-25T18:44:44.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.957+0000 7f132d600700 1 -- 172.21.15.63:0/4133553619 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1324013070 con 0x7f133006a750 2024-04-25T18:44:44.959 INFO:teuthology.orchestra.run.smithi063.stdout:176093659217 2024-04-25T18:44:44.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.960+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 msgr2=0x7f13180a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:44.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 0x7f13180a6180 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1330102f00 tx=0x7f131c009260 comp rx=0 tx=0).stop 2024-04-25T18:44:44.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 msgr2=0x7f133019b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:44.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13240057f0 tx=0x7f132400bfd0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 shutdown_connections 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f13180a3cc0 0x7f13180a6180 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f133006a750 0x7f133019b330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f133010f4f0 0x7f133019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 --2- 172.21.15.63:0/4133553619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f133006b070 0x7f133019b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 >> 172.21.15.63:0/4133553619 conn(0x7f13300788f0 msgr2=0x7f13301109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 shutdown_connections 2024-04-25T18:44:44.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.961+0000 7f1337600700 1 -- 172.21.15.63:0/4133553619 wait complete. 2024-04-25T18:44:45.124 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:44 smithi063 ceph-mon[28006]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:44 smithi070 ceph-mon[32820]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:44 smithi114 ceph-mon[32988]: pgmap v411: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:45.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738470 got 34359738477 for osd.0 2024-04-25T18:44:45.340 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:46.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:45 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3993899305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T18:44:46.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:45 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4133553619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T18:44:46.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 -- 172.21.15.63:0/670640564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 msgr2=0x7f626c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 --2- 172.21.15.63:0/670640564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c102f70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6258009920 tx=0x7f6258009c30 comp rx=0 tx=0).stop 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 -- 172.21.15.63:0/670640564 shutdown_connections 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 --2- 172.21.15.63:0/670640564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 --2- 172.21.15.63:0/670640564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 --2- 172.21.15.63:0/670640564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c102f70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 -- 172.21.15.63:0/670640564 >> 172.21.15.63:0/670640564 conn(0x7f626c079ba0 msgr2=0x7f626c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 -- 172.21.15.63:0/670640564 shutdown_connections 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.592+0000 7f6271200700 1 -- 172.21.15.63:0/670640564 wait complete. 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.594+0000 7f6271200700 1 Processor -- start 2024-04-25T18:44:46.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.594+0000 7f6271200700 1 -- start start 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c19da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f626c112c30 con 0x7f626c1034b0 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f626c112ab0 con 0x7f626c102af0 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6271200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f626c112db0 con 0x7f626c107940 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6269e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6269400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6269e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57476/0 (socket says 172.21.15.63:57476) 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f6269e00700 1 -- 172.21.15.63:0/3993899305 learned_addr learned my addr 172.21.15.63:0/3993899305 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.595+0000 7f626a800700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c19da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 -- 172.21.15.63:0/3993899305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 msgr2=0x7f626c19da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:46.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c19da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 -- 172.21.15.63:0/3993899305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 msgr2=0x7f626c1991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 -- 172.21.15.63:0/3993899305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626c19e1b0 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269e00700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f626a800700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c19da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6269400700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c1996e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6258000c00 tx=0x7f625800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258027070 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626c19e440 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.596+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6258005670 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.597+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62580049c0 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.597+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626c074c70 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.598+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626c069730 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.602+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f625802a070 con 0x7f626c1034b0 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.603+0000 7f6262a00700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 0x7f62500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:46.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.604+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f62580d1210 con 0x7f626c1034b0 2024-04-25T18:44:46.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.604+0000 7f6269e00700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 0x7f62500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:46.161 INFO:teuthology.orchestra.run.smithi063.stdout:270582939712 2024-04-25T18:44:46.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.605+0000 7f6269e00700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 0x7f62500a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f625c005950 tx=0x7f625c00b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:46.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.605+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6258031080 con 0x7f626c1034b0 2024-04-25T18:44:46.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.923+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f626c19a500 con 0x7f626c1034b0 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.924+0000 7f6262a00700 1 -- 172.21.15.63:0/3993899305 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6258013030 con 0x7f626c1034b0 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 msgr2=0x7f62500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 0x7f62500a6100 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f625c005950 tx=0x7f625c00b500 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 msgr2=0x7f626c1996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c1996e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6258000c00 tx=0x7f625800fa00 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 shutdown_connections 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f62500a3c40 0x7f62500a6100 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f626c107940 0x7f626c19da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f626c102af0 0x7f626c1991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 --2- 172.21.15.63:0/3993899305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f626c1034b0 0x7f626c1996e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 >> 172.21.15.63:0/3993899305 conn(0x7f626c079ba0 msgr2=0x7f626c100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 shutdown_connections 2024-04-25T18:44:46.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:44.928+0000 7f6271200700 1 -- 172.21.15.63:0/3993899305 wait complete. 2024-04-25T18:44:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:45 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3993899305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T18:44:46.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:45 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4133553619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T18:44:46.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:45 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3993899305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T18:44:46.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:45 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4133553619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T18:44:47.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678067 got 304942678071 for osd.9 2024-04-25T18:44:47.041 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:46 smithi063 ceph-mon[28006]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:46 smithi070 ceph-mon[32820]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:47.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:46 smithi114 ceph-mon[32988]: pgmap v412: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:48.795 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659211 got 176093659217 for osd.5 2024-04-25T18:44:48.796 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:49.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.040+0000 7fa8c2800700 1 -- 172.21.15.63:0/1276199260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 msgr2=0x7fa8bc111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:49.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.040+0000 7fa8c2800700 1 --2- 172.21.15.63:0/1276199260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc111890 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400b0a0 tx=0x7fa8b400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 -- 172.21.15.63:0/1276199260 shutdown_connections 2024-04-25T18:44:49.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 --2- 172.21.15.63:0/1276199260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc111890 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 --2- 172.21.15.63:0/1276199260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc074c70 0x7fa8bc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 --2- 172.21.15.63:0/1276199260 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc075590 0x7fa8bc10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 -- 172.21.15.63:0/1276199260 >> 172.21.15.63:0/1276199260 conn(0x7fa8bc079ba0 msgr2=0x7fa8bc079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:49.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 -- 172.21.15.63:0/1276199260 shutdown_connections 2024-04-25T18:44:49.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.041+0000 7fa8c2800700 1 -- 172.21.15.63:0/1276199260 wait complete. 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.042+0000 7fa8c2800700 1 Processor -- start 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.042+0000 7fa8c2800700 1 -- start start 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.042+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc074c70 0x7fa8bc19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.042+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8c2800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8c2800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc114e80 con 0x7fa8bc10f4a0 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8c2800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc114d00 con 0x7fa8bc075590 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8c2800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa8bc115000 con 0x7fa8bc074c70 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:49.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8bb400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc074c70 0x7fa8bc19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:49.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53784/0 (socket says 172.21.15.63:53784) 2024-04-25T18:44:49.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 -- 172.21.15.63:0/3181088493 learned_addr learned my addr 172.21.15.63:0/3181088493 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:49.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8bbe00700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 -- 172.21.15.63:0/3181088493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc074c70 msgr2=0x7fa8bc19b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc074c70 0x7fa8bc19b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 -- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 msgr2=0x7fa8bc19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.043+0000 7fa8b2a00700 1 -- 172.21.15.63:0/3181088493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8bc1a0300 con 0x7fa8bc075590 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8b2a00700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400e9c0 tx=0x7fa8a400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400cbd0 con 0x7fa8bc075590 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a400ee60 con 0x7fa8bc075590 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40055c0 con 0x7fa8bc075590 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8bc1a05f0 con 0x7fa8bc075590 2024-04-25T18:44:49.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.044+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8bc1a7e70 con 0x7fa8bc075590 2024-04-25T18:44:49.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.045+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8bc069730 con 0x7fa8bc075590 2024-04-25T18:44:49.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.047+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa8a4004750 con 0x7fa8bc075590 2024-04-25T18:44:49.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.048+0000 7fa8b8c00700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 0x7fa8a00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:49.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.048+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fa8a40d0670 con 0x7fa8bc075590 2024-04-25T18:44:49.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.049+0000 7fa8bb400700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 0x7fa8a00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:49.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.049+0000 7fa8bb400700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 0x7fa8a00a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009680 tx=0x7fa8ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:49.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.053+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8a409cfc0 con 0x7fa8bc075590 2024-04-25T18:44:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:48 smithi063 ceph-mon[28006]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:49.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:48 smithi070 ceph-mon[32820]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:49.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.289+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa8bc02df80 con 0x7fa8bc075590 2024-04-25T18:44:49.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.290+0000 7fa8b8c00700 1 -- 172.21.15.63:0/3181088493 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa8a409c8c0 con 0x7fa8bc075590 2024-04-25T18:44:49.291 INFO:teuthology.orchestra.run.smithi063.stdout:77309411428 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 msgr2=0x7fa8a00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 0x7fa8a00a6180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac009680 tx=0x7fa8ac009260 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 msgr2=0x7fa8bc19b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa8a400e9c0 tx=0x7fa8a400ecd0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 shutdown_connections 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa8a00a3cc0 0x7fa8a00a6180 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa8bc074c70 0x7fa8bc19b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa8bc10f4a0 0x7fa8bc19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 --2- 172.21.15.63:0/3181088493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8bc075590 0x7fa8bc19b830 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 >> 172.21.15.63:0/3181088493 conn(0x7fa8bc079ba0 msgr2=0x7fa8bc0ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 shutdown_connections 2024-04-25T18:44:49.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:49.293+0000 7fa8c2800700 1 -- 172.21.15.63:0/3181088493 wait complete. 2024-04-25T18:44:49.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:48 smithi114 ceph-mon[32988]: pgmap v413: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:50.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:49 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3181088493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T18:44:50.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:49 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3181088493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T18:44:50.323 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:49 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3181088493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T18:44:50.496 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411420 got 77309411428 for osd.2 2024-04-25T18:44:50.496 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:51.118 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939706 got 270582939712 for osd.8 2024-04-25T18:44:51.118 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:51.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:50 smithi070 ceph-mon[32820]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:51.238 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:50 smithi063 ceph-mon[28006]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:50 smithi114 ceph-mon[32988]: pgmap v414: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 -- 172.21.15.63:0/1964717391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 msgr2=0x7f78681096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 --2- 172.21.15.63:0/1964717391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f78681096b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f785c009990 tx=0x7f785c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 -- 172.21.15.63:0/1964717391 shutdown_connections 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 --2- 172.21.15.63:0/1964717391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f786810e3b0 0x7f7868115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 --2- 172.21.15.63:0/1964717391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f78681096b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 --2- 172.21.15.63:0/1964717391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7868109c80 0x7f786810dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 -- 172.21.15.63:0/1964717391 >> 172.21.15.63:0/1964717391 conn(0x7f7868076210 msgr2=0x7f7868107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 -- 172.21.15.63:0/1964717391 shutdown_connections 2024-04-25T18:44:51.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.414+0000 7f7870600700 1 -- 172.21.15.63:0/1964717391 wait complete. 2024-04-25T18:44:51.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.415+0000 7f7870600700 1 Processor -- start 2024-04-25T18:44:51.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 -- start start 2024-04-25T18:44:51.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868109c80 0x7f786819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f786810e3b0 0x7f78681a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7868119290 con 0x7f786810e3b0 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7868119110 con 0x7f78681092d0 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f7870600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7868119410 con 0x7f7868109c80 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f786dc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f786d200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868109c80 0x7f786819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f786dc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53796/0 (socket says 172.21.15.63:53796) 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f786dc00700 1 -- 172.21.15.63:0/1439384353 learned_addr learned my addr 172.21.15.63:0/1439384353 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:51.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.416+0000 7f786e600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f786810e3b0 0x7f78681a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:51.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 -- 172.21.15.63:0/1439384353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868109c80 msgr2=0x7f786819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868109c80 0x7f786819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 -- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f786810e3b0 msgr2=0x7f78681a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f786810e3b0 0x7f78681a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 -- 172.21.15.63:0/1439384353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78681a4730 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.417+0000 7f786dc00700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f785c005910 tx=0x7f785c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.418+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c027070 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.418+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f785c00b610 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.418+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78681a49c0 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.418+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0173f0 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.418+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78681ac2a0 con 0x7f78681092d0 2024-04-25T18:44:51.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.419+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7868069730 con 0x7f78681092d0 2024-04-25T18:44:51.422 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.421+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f785c02a070 con 0x7f78681092d0 2024-04-25T18:44:51.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.422+0000 7f7866000700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 0x7f78500a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:51.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.422+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f785c0d1090 con 0x7f78681092d0 2024-04-25T18:44:51.423 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.422+0000 7f786d200700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 0x7f78500a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:51.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.423+0000 7f786d200700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 0x7f78500a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f786810dde0 tx=0x7f78540073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:51.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.425+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f785c013070 con 0x7f78681092d0 2024-04-25T18:44:51.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 360777252901 got 360777252909 for osd.11 2024-04-25T18:44:51.620 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:51.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.664+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f786802df80 con 0x7f78681092d0 2024-04-25T18:44:51.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.665+0000 7f7866000700 1 -- 172.21.15.63:0/1439384353 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f785c09dad0 con 0x7f78681092d0 2024-04-25T18:44:51.666 INFO:teuthology.orchestra.run.smithi063.stdout:326417514548 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 msgr2=0x7f78500a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 0x7f78500a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f786810dde0 tx=0x7f78540073d0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 msgr2=0x7f786819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f785c005910 tx=0x7f785c0047b0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 shutdown_connections 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f78500a3cc0 0x7f78500a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f786810e3b0 0x7f78681a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78681092d0 0x7f786819f720 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 --2- 172.21.15.63:0/1439384353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7868109c80 0x7f786819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 >> 172.21.15.63:0/1439384353 conn(0x7f7868076210 msgr2=0x7f7868077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:51.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.668+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 shutdown_connections 2024-04-25T18:44:51.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:51.669+0000 7f7870600700 1 -- 172.21.15.63:0/1439384353 wait complete. 2024-04-25T18:44:52.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:51 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1439384353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T18:44:52.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:51 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1439384353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T18:44:52.345 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:51 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1439384353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T18:44:52.774 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514542 got 326417514548 for osd.10 2024-04-25T18:44:52.774 DEBUG:teuthology.parallel:result is None 2024-04-25T18:44:52.774 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T18:44:52.774 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:44:52.996 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:52 smithi063 ceph-mon[28006]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:53.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:52 smithi070 ceph-mon[32820]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:53.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:52 smithi114 ceph-mon[32988]: pgmap v415: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:53.363 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:55.172 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:54 smithi063 ceph-mon[28006]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:55.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:54 smithi070 ceph-mon[32820]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:54 smithi114 ceph-mon[32988]: pgmap v416: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.721+0000 7fd141000700 1 -- 172.21.15.63:0/3354883582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c102a90 msgr2=0x7fd13c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.721+0000 7fd141000700 1 --2- 172.21.15.63:0/3354883582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c102a90 0x7fd13c102f10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1280098f0 tx=0x7fd128009c00 comp rx=0 tx=0).stop 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 -- 172.21.15.63:0/3354883582 shutdown_connections 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 --2- 172.21.15.63:0/3354883582 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 0x7fd13c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 --2- 172.21.15.63:0/3354883582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c108a80 0x7fd13c108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:55.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 --2- 172.21.15.63:0/3354883582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c102a90 0x7fd13c102f10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:55.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 -- 172.21.15.63:0/3354883582 >> 172.21.15.63:0/3354883582 conn(0x7fd13c079bf0 msgr2=0x7fd13c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:55.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 -- 172.21.15.63:0/3354883582 shutdown_connections 2024-04-25T18:44:55.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.722+0000 7fd141000700 1 -- 172.21.15.63:0/3354883582 wait complete. 2024-04-25T18:44:55.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.723+0000 7fd141000700 1 Processor -- start 2024-04-25T18:44:55.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.723+0000 7fd141000700 1 -- start start 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 0x7fd13c19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 0x7fd13c19b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 0x7fd13c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c112c50 con 0x7fd13c102a90 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c112ad0 con 0x7fd13c108a80 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd141000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c112dd0 con 0x7fd13c103450 2024-04-25T18:44:55.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd139e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 0x7fd13c19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:55.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd139e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 0x7fd13c19b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55620/0 (socket says 172.21.15.63:55620) 2024-04-25T18:44:55.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd139e00700 1 -- 172.21.15.63:0/747075104 learned_addr learned my addr 172.21.15.63:0/747075104 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:55.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd13a800700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 0x7fd13c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:55.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.724+0000 7fd139400700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 0x7fd13c19b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 -- 172.21.15.63:0/747075104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 msgr2=0x7fd13c19b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 0x7fd13c19b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 -- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 msgr2=0x7fd13c19b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 0x7fd13c19b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 -- 172.21.15.63:0/747075104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13c1a03f0 con 0x7fd13c108a80 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd13a800700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 0x7fd13c19fcb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd13400efc0 tx=0x7fd13400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1340092e0 con 0x7fd13c108a80 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13c1a0680 con 0x7fd13c108a80 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd134004bb0 con 0x7fd13c108a80 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.725+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134004020 con 0x7fd13c108a80 2024-04-25T18:44:55.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.726+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13c1a7f60 con 0x7fd13c108a80 2024-04-25T18:44:55.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.727+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd13c1041b0 con 0x7fd13c108a80 2024-04-25T18:44:55.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.729+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd13401b030 con 0x7fd13c108a80 2024-04-25T18:44:55.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.730+0000 7fd132a00700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 0x7fd1200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:55.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.730+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fd1340cfdb0 con 0x7fd13c108a80 2024-04-25T18:44:55.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.730+0000 7fd139e00700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 0x7fd1200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:55.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.731+0000 7fd139e00700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 0x7fd1200a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009750 tx=0x7fd12c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:55.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.735+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd134022020 con 0x7fd13c108a80 2024-04-25T18:44:55.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.989+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd13c10f650 con 0x7fd1200a3c40 2024-04-25T18:44:56.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:55.999+0000 7fd132a00700 1 -- 172.21.15.63:0/747075104 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258850 (secure 0 0 0) 0x7fd13c10f650 con 0x7fd1200a3c40 2024-04-25T18:44:56.006 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:44:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.004+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 msgr2=0x7fd1200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.004+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 0x7fd1200a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009750 tx=0x7fd12c009310 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 msgr2=0x7fd13c19fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 0x7fd13c19fcb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd13400efc0 tx=0x7fd13400c3a0 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 shutdown_connections 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd1200a3c40 0x7fd1200a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd13c108a80 0x7fd13c19fcb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd13c102a90 0x7fd13c19b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 --2- 172.21.15.63:0/747075104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd13c103450 0x7fd13c19b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 >> 172.21.15.63:0/747075104 conn(0x7fd13c079bf0 msgr2=0x7fd13c10df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 shutdown_connections 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:56.005+0000 7fd141000700 1 -- 172.21.15.63:0/747075104 wait complete. 2024-04-25T18:44:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:44:56.826 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-04-25T18:44:55.186985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654755,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":878,"num_read_kb":555,"num_write":354,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392732,"kb_used_data":14460,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549092,"statfs":{"total":1151940427776,"available":1151538270208,"internally_reserved":0,"allocated":14807040,"data_stored":8124441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005351"},"pg_stats":[{"pgid":"3.3f","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115736+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:44:23.115736+0000","last_peered":"2024-04-25T18:44:23.115736+0000","last_clean":"2024-04-25T18:44:23.115736+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:44:23.115736+0000","last_undegraded":"2024-04-25T18:44:23.115736+0000","last_fullsized":"2024-04-25T18:44:23.115736+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"96'4","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.112259+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:44:23.112259+0000","last_peered":"2024-04-25T18:44:23.112259+0000","last_clean":"2024-04-25T18:44:23.112259+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:44:23.112259+0000","last_undegraded":"2024-04-25T18:44:23.112259+0000","last_fullsized":"2024-04-25T18:44:23.112259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.117310+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:44:23.117310+0000","last_peered":"2024-04-25T18:44:23.117310+0000","last_clean":"2024-04-25T18:44:23.117310+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:44:23.117310+0000","last_undegraded":"2024-04-25T18:44:23.117310+0000","last_fullsized":"2024-04-25T18:44:23.117310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.098636+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:44:23.098636+0000","last_peered":"2024-04-25T18:44:23.098636+0000","last_clean":"2024-04-25T18:44:23.098636+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:44:23.098636+0000","last_undegraded":"2024-04-25T18:44:23.098636+0000","last_fullsized":"2024-04-25T18:44:23.098636+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"96'4","reported_seq":28,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101893+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:44:23.101893+0000","last_peered":"2024-04-25T18:44:23.101893+0000","last_clean":"2024-04-25T18:44:23.101893+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:44:23.101893+0000","last_undegraded":"2024-04-25T18:44:23.101893+0000","last_fullsized":"2024-04-25T18:44:23.101893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"96'6","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102130+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:44:23.102130+0000","last_peered":"2024-04-25T18:44:23.102130+0000","last_clean":"2024-04-25T18:44:23.102130+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:44:23.102130+0000","last_undegraded":"2024-04-25T18:44:23.102130+0000","last_fullsized":"2024-04-25T18:44:23.102130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"96'5","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.106962+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:44:23.106962+0000","last_peered":"2024-04-25T18:44:23.106962+0000","last_clean":"2024-04-25T18:44:23.106962+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:44:23.106962+0000","last_undegraded":"2024-04-25T18:44:23.106962+0000","last_fullsized":"2024-04-25T18:44:23.106962+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'9","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.122191+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:44:23.122191+0000","last_peered":"2024-04-25T18:44:23.122191+0000","last_clean":"2024-04-25T18:44:23.122191+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:44:23.122191+0000","last_undegraded":"2024-04-25T18:44:23.122191+0000","last_fullsized":"2024-04-25T18:44:23.122191+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'11","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.112737+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:44:23.112737+0000","last_peered":"2024-04-25T18:44:23.112737+0000","last_clean":"2024-04-25T18:44:23.112737+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:44:23.112737+0000","last_undegraded":"2024-04-25T18:44:23.112737+0000","last_fullsized":"2024-04-25T18:44:23.112737+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"96'16","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103435+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:44:23.103435+0000","last_peered":"2024-04-25T18:44:23.103435+0000","last_clean":"2024-04-25T18:44:23.103435+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:44:23.103435+0000","last_undegraded":"2024-04-25T18:44:23.103435+0000","last_fullsized":"2024-04-25T18:44:23.103435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115432+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:44:23.115432+0000","last_peered":"2024-04-25T18:44:23.115432+0000","last_clean":"2024-04-25T18:44:23.115432+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:44:23.115432+0000","last_undegraded":"2024-04-25T18:44:23.115432+0000","last_fullsized":"2024-04-25T18:44:23.115432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.093830+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:44:23.093830+0000","last_peered":"2024-04-25T18:44:23.093830+0000","last_clean":"2024-04-25T18:44:23.093830+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:44:23.093830+0000","last_undegraded":"2024-04-25T18:44:23.093830+0000","last_fullsized":"2024-04-25T18:44:23.093830+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'10","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.106779+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:44:23.106779+0000","last_peered":"2024-04-25T18:44:23.106779+0000","last_clean":"2024-04-25T18:44:23.106779+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:44:23.106779+0000","last_undegraded":"2024-04-25T18:44:23.106779+0000","last_fullsized":"2024-04-25T18:44:23.106779+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.113639+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:44:23.113639+0000","last_peered":"2024-04-25T18:44:23.113639+0000","last_clean":"2024-04-25T18:44:23.113639+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:44:23.113639+0000","last_undegraded":"2024-04-25T18:44:23.113639+0000","last_fullsized":"2024-04-25T18:44:23.113639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115349+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:44:23.115349+0000","last_peered":"2024-04-25T18:44:23.115349+0000","last_clean":"2024-04-25T18:44:23.115349+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:44:23.115349+0000","last_undegraded":"2024-04-25T18:44:23.115349+0000","last_fullsized":"2024-04-25T18:44:23.115349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119296+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:44:23.119296+0000","last_peered":"2024-04-25T18:44:23.119296+0000","last_clean":"2024-04-25T18:44:23.119296+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:44:23.119296+0000","last_undegraded":"2024-04-25T18:44:23.119296+0000","last_fullsized":"2024-04-25T18:44:23.119296+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115732+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:44:23.115732+0000","last_peered":"2024-04-25T18:44:23.115732+0000","last_clean":"2024-04-25T18:44:23.115732+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:44:23.115732+0000","last_undegraded":"2024-04-25T18:44:23.115732+0000","last_fullsized":"2024-04-25T18:44:23.115732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121838+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:44:23.121838+0000","last_peered":"2024-04-25T18:44:23.121838+0000","last_clean":"2024-04-25T18:44:23.121838+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:44:23.121838+0000","last_undegraded":"2024-04-25T18:44:23.121838+0000","last_fullsized":"2024-04-25T18:44:23.121838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.120929+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:44:23.120929+0000","last_peered":"2024-04-25T18:44:23.120929+0000","last_clean":"2024-04-25T18:44:23.120929+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:44:23.120929+0000","last_undegraded":"2024-04-25T18:44:23.120929+0000","last_fullsized":"2024-04-25T18:44:23.120929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.359876+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:43:55.359876+0000","last_peered":"2024-04-25T18:43:55.359876+0000","last_clean":"2024-04-25T18:43:55.359876+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:43:55.359876+0000","last_undegraded":"2024-04-25T18:43:55.359876+0000","last_fullsized":"2024-04-25T18:43:55.359876+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119893+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:44:23.119893+0000","last_peered":"2024-04-25T18:44:23.119893+0000","last_clean":"2024-04-25T18:44:23.119893+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:44:23.119893+0000","last_undegraded":"2024-04-25T18:44:23.119893+0000","last_fullsized":"2024-04-25T18:44:23.119893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.428032+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:43:55.428032+0000","last_peered":"2024-04-25T18:43:55.428032+0000","last_clean":"2024-04-25T18:43:55.428032+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:43:55.428032+0000","last_undegraded":"2024-04-25T18:43:55.428032+0000","last_fullsized":"2024-04-25T18:43:55.428032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121560+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:44:23.121560+0000","last_peered":"2024-04-25T18:44:23.121560+0000","last_clean":"2024-04-25T18:44:23.121560+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:44:23.121560+0000","last_undegraded":"2024-04-25T18:44:23.121560+0000","last_fullsized":"2024-04-25T18:44:23.121560+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423737+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:43:55.423737+0000","last_peered":"2024-04-25T18:43:55.423737+0000","last_clean":"2024-04-25T18:43:55.423737+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:43:55.423737+0000","last_undegraded":"2024-04-25T18:43:55.423737+0000","last_fullsized":"2024-04-25T18:43:55.423737+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"96'9","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123522+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:44:23.123522+0000","last_peered":"2024-04-25T18:44:23.123522+0000","last_clean":"2024-04-25T18:44:23.123522+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:44:23.123522+0000","last_undegraded":"2024-04-25T18:44:23.123522+0000","last_fullsized":"2024-04-25T18:44:23.123522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423951+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:43:55.423951+0000","last_peered":"2024-04-25T18:43:55.423951+0000","last_clean":"2024-04-25T18:43:55.423951+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:43:55.423951+0000","last_undegraded":"2024-04-25T18:43:55.423951+0000","last_fullsized":"2024-04-25T18:43:55.423951+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"96'15","reported_seq":45,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.105593+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:44:23.105593+0000","last_peered":"2024-04-25T18:44:23.105593+0000","last_clean":"2024-04-25T18:44:23.105593+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:44:23.105593+0000","last_undegraded":"2024-04-25T18:44:23.105593+0000","last_fullsized":"2024-04-25T18:44:23.105593+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":157,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:51.796243+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:44:51.796243+0000","last_peered":"2024-04-25T18:44:51.796243+0000","last_clean":"2024-04-25T18:44:51.796243+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:44:51.796243+0000","last_undegraded":"2024-04-25T18:44:51.796243+0000","last_fullsized":"2024-04-25T18:44:51.796243+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"96'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103869+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:44:23.103869+0000","last_peered":"2024-04-25T18:44:23.103869+0000","last_clean":"2024-04-25T18:44:23.103869+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:44:23.103869+0000","last_undegraded":"2024-04-25T18:44:23.103869+0000","last_fullsized":"2024-04-25T18:44:23.103869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.350478+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:43:55.350478+0000","last_peered":"2024-04-25T18:43:55.350478+0000","last_clean":"2024-04-25T18:43:55.350478+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:43:55.350478+0000","last_undegraded":"2024-04-25T18:43:55.350478+0000","last_fullsized":"2024-04-25T18:43:55.350478+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"96'8","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121308+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:44:23.121308+0000","last_peered":"2024-04-25T18:44:23.121308+0000","last_clean":"2024-04-25T18:44:23.121308+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:44:23.121308+0000","last_undegraded":"2024-04-25T18:44:23.121308+0000","last_fullsized":"2024-04-25T18:44:23.121308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.354051+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:43:55.354051+0000","last_peered":"2024-04-25T18:43:55.354051+0000","last_clean":"2024-04-25T18:43:55.354051+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:43:55.354051+0000","last_undegraded":"2024-04-25T18:43:55.354051+0000","last_fullsized":"2024-04-25T18:43:55.354051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123009+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:44:23.123009+0000","last_peered":"2024-04-25T18:44:23.123009+0000","last_clean":"2024-04-25T18:44:23.123009+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:44:23.123009+0000","last_undegraded":"2024-04-25T18:44:23.123009+0000","last_fullsized":"2024-04-25T18:44:23.123009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423469+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:43:55.423469+0000","last_peered":"2024-04-25T18:43:55.423469+0000","last_clean":"2024-04-25T18:43:55.423469+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:43:55.423469+0000","last_undegraded":"2024-04-25T18:43:55.423469+0000","last_fullsized":"2024-04-25T18:43:55.423469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'11","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121584+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:44:23.121584+0000","last_peered":"2024-04-25T18:44:23.121584+0000","last_clean":"2024-04-25T18:44:23.121584+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:44:23.121584+0000","last_undegraded":"2024-04-25T18:44:23.121584+0000","last_fullsized":"2024-04-25T18:44:23.121584+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":153,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:51.795560+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:44:51.795560+0000","last_peered":"2024-04-25T18:44:51.795560+0000","last_clean":"2024-04-25T18:44:51.795560+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:44:51.795560+0000","last_undegraded":"2024-04-25T18:44:51.795560+0000","last_fullsized":"2024-04-25T18:44:51.795560+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"96'10","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.122679+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:44:23.122679+0000","last_peered":"2024-04-25T18:44:23.122679+0000","last_clean":"2024-04-25T18:44:23.122679+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:44:23.122679+0000","last_undegraded":"2024-04-25T18:44:23.122679+0000","last_fullsized":"2024-04-25T18:44:23.122679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.434184+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:43:55.434184+0000","last_peered":"2024-04-25T18:43:55.434184+0000","last_clean":"2024-04-25T18:43:55.434184+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:43:55.434184+0000","last_undegraded":"2024-04-25T18:43:55.434184+0000","last_fullsized":"2024-04-25T18:43:55.434184+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"96'9","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.116578+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:44:23.116578+0000","last_peered":"2024-04-25T18:44:23.116578+0000","last_clean":"2024-04-25T18:44:23.116578+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:44:23.116578+0000","last_undegraded":"2024-04-25T18:44:23.116578+0000","last_fullsized":"2024-04-25T18:44:23.116578+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423919+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:43:55.423919+0000","last_peered":"2024-04-25T18:43:55.423919+0000","last_clean":"2024-04-25T18:43:55.423919+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:43:55.423919+0000","last_undegraded":"2024-04-25T18:43:55.423919+0000","last_fullsized":"2024-04-25T18:43:55.423919+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"96'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119652+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:44:23.119652+0000","last_peered":"2024-04-25T18:44:23.119652+0000","last_clean":"2024-04-25T18:44:23.119652+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:44:23.119652+0000","last_undegraded":"2024-04-25T18:44:23.119652+0000","last_fullsized":"2024-04-25T18:44:23.119652+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"96'32","reported_seq":311,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:51.795820+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:44:51.795820+0000","last_peered":"2024-04-25T18:44:51.795820+0000","last_clean":"2024-04-25T18:44:51.795820+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:44:51.795820+0000","last_undegraded":"2024-04-25T18:44:51.795820+0000","last_fullsized":"2024-04-25T18:44:51.795820+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60011,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":248,"num_read_kb":182,"num_write":33,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"96'6","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.098940+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:44:23.098940+0000","last_peered":"2024-04-25T18:44:23.098940+0000","last_clean":"2024-04-25T18:44:23.098940+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:44:23.098940+0000","last_undegraded":"2024-04-25T18:44:23.098940+0000","last_fullsized":"2024-04-25T18:44:23.098940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351253+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:43:55.351253+0000","last_peered":"2024-04-25T18:43:55.351253+0000","last_clean":"2024-04-25T18:43:55.351253+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:43:55.351253+0000","last_undegraded":"2024-04-25T18:43:55.351253+0000","last_fullsized":"2024-04-25T18:43:55.351253+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.124289+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:44:23.124289+0000","last_peered":"2024-04-25T18:44:23.124289+0000","last_clean":"2024-04-25T18:44:23.124289+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:44:23.124289+0000","last_undegraded":"2024-04-25T18:44:23.124289+0000","last_fullsized":"2024-04-25T18:44:23.124289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423445+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:43:55.423445+0000","last_peered":"2024-04-25T18:43:55.423445+0000","last_clean":"2024-04-25T18:43:55.423445+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:43:55.423445+0000","last_undegraded":"2024-04-25T18:43:55.423445+0000","last_fullsized":"2024-04-25T18:43:55.423445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'15","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.118162+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:44:23.118162+0000","last_peered":"2024-04-25T18:44:23.118162+0000","last_clean":"2024-04-25T18:44:23.118162+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:44:23.118162+0000","last_undegraded":"2024-04-25T18:44:23.118162+0000","last_fullsized":"2024-04-25T18:44:23.118162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"96'5","reported_seq":160,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:51.870347+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:44:51.870347+0000","last_peered":"2024-04-25T18:44:51.870347+0000","last_clean":"2024-04-25T18:44:51.870347+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:44:51.870347+0000","last_undegraded":"2024-04-25T18:44:51.870347+0000","last_fullsized":"2024-04-25T18:44:51.870347+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101174+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:44:23.101174+0000","last_peered":"2024-04-25T18:44:23.101174+0000","last_clean":"2024-04-25T18:44:23.101174+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:44:23.101174+0000","last_undegraded":"2024-04-25T18:44:23.101174+0000","last_fullsized":"2024-04-25T18:44:23.101174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.434037+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:43:55.434037+0000","last_peered":"2024-04-25T18:43:55.434037+0000","last_clean":"2024-04-25T18:43:55.434037+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:43:55.434037+0000","last_undegraded":"2024-04-25T18:43:55.434037+0000","last_fullsized":"2024-04-25T18:43:55.434037+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.105260+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:44:23.105260+0000","last_peered":"2024-04-25T18:44:23.105260+0000","last_clean":"2024-04-25T18:44:23.105260+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:44:23.105260+0000","last_undegraded":"2024-04-25T18:44:23.105260+0000","last_fullsized":"2024-04-25T18:44:23.105260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423704+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:43:55.423704+0000","last_peered":"2024-04-25T18:43:55.423704+0000","last_clean":"2024-04-25T18:43:55.423704+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:43:55.423704+0000","last_undegraded":"2024-04-25T18:43:55.423704+0000","last_fullsized":"2024-04-25T18:43:55.423704+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103933+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:44:23.103933+0000","last_peered":"2024-04-25T18:44:23.103933+0000","last_clean":"2024-04-25T18:44:23.103933+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:44:23.103933+0000","last_undegraded":"2024-04-25T18:44:23.103933+0000","last_fullsized":"2024-04-25T18:44:23.103933+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.359384+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:43:55.359384+0000","last_peered":"2024-04-25T18:43:55.359384+0000","last_clean":"2024-04-25T18:43:55.359384+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:43:55.359384+0000","last_undegraded":"2024-04-25T18:43:55.359384+0000","last_fullsized":"2024-04-25T18:43:55.359384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123292+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:44:23.123292+0000","last_peered":"2024-04-25T18:44:23.123292+0000","last_clean":"2024-04-25T18:44:23.123292+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:44:23.123292+0000","last_undegraded":"2024-04-25T18:44:23.123292+0000","last_fullsized":"2024-04-25T18:44:23.123292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423474+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:43:55.423474+0000","last_peered":"2024-04-25T18:43:55.423474+0000","last_clean":"2024-04-25T18:43:55.423474+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:43:55.423474+0000","last_undegraded":"2024-04-25T18:43:55.423474+0000","last_fullsized":"2024-04-25T18:43:55.423474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.114766+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:44:23.114766+0000","last_peered":"2024-04-25T18:44:23.114766+0000","last_clean":"2024-04-25T18:44:23.114766+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:44:23.114766+0000","last_undegraded":"2024-04-25T18:44:23.114766+0000","last_fullsized":"2024-04-25T18:44:23.114766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"96'8","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115302+0000","last_change":"2024-04-25T18:43:55.423277+0000","last_active":"2024-04-25T18:44:23.115302+0000","last_peered":"2024-04-25T18:44:23.115302+0000","last_clean":"2024-04-25T18:44:23.115302+0000","last_became_active":"2024-04-25T18:43:55.421960+0000","last_became_peered":"2024-04-25T18:43:55.421960+0000","last_unstale":"2024-04-25T18:44:23.115302+0000","last_undegraded":"2024-04-25T18:44:23.115302+0000","last_fullsized":"2024-04-25T18:44:23.115302+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:27.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"96'5","reported_seq":155,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:50.867398+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:44:50.867398+0000","last_peered":"2024-04-25T18:44:50.867398+0000","last_clean":"2024-04-25T18:44:50.867398+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:44:50.867398+0000","last_undegraded":"2024-04-25T18:44:50.867398+0000","last_fullsized":"2024-04-25T18:44:50.867398+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":119,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123069+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:44:23.123069+0000","last_peered":"2024-04-25T18:44:23.123069+0000","last_clean":"2024-04-25T18:44:23.123069+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:44:23.123069+0000","last_undegraded":"2024-04-25T18:44:23.123069+0000","last_fullsized":"2024-04-25T18:44:23.123069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.420012+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:43:55.420012+0000","last_peered":"2024-04-25T18:43:55.420012+0000","last_clean":"2024-04-25T18:43:55.420012+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:43:55.420012+0000","last_undegraded":"2024-04-25T18:43:55.420012+0000","last_fullsized":"2024-04-25T18:43:55.420012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423348+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:43:55.423348+0000","last_peered":"2024-04-25T18:43:55.423348+0000","last_clean":"2024-04-25T18:43:55.423348+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:43:55.423348+0000","last_undegraded":"2024-04-25T18:43:55.423348+0000","last_fullsized":"2024-04-25T18:43:55.423348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102301+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:44:23.102301+0000","last_peered":"2024-04-25T18:44:23.102301+0000","last_clean":"2024-04-25T18:44:23.102301+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:44:23.102301+0000","last_undegraded":"2024-04-25T18:44:23.102301+0000","last_fullsized":"2024-04-25T18:44:23.102301+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423588+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:43:55.423588+0000","last_peered":"2024-04-25T18:43:55.423588+0000","last_clean":"2024-04-25T18:43:55.423588+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:43:55.423588+0000","last_undegraded":"2024-04-25T18:43:55.423588+0000","last_fullsized":"2024-04-25T18:43:55.423588+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423631+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:43:55.423631+0000","last_peered":"2024-04-25T18:43:55.423631+0000","last_clean":"2024-04-25T18:43:55.423631+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:43:55.423631+0000","last_undegraded":"2024-04-25T18:43:55.423631+0000","last_fullsized":"2024-04-25T18:43:55.423631+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.112759+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:44:23.112759+0000","last_peered":"2024-04-25T18:44:23.112759+0000","last_clean":"2024-04-25T18:44:23.112759+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:44:23.112759+0000","last_undegraded":"2024-04-25T18:44:23.112759+0000","last_fullsized":"2024-04-25T18:44:23.112759+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"96'12","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.918564+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:44:23.918564+0000","last_peered":"2024-04-25T18:44:23.918564+0000","last_clean":"2024-04-25T18:44:23.918564+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:44:23.918564+0000","last_undegraded":"2024-04-25T18:44:23.918564+0000","last_fullsized":"2024-04-25T18:44:23.918564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123012+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:44:23.123012+0000","last_peered":"2024-04-25T18:44:23.123012+0000","last_clean":"2024-04-25T18:44:23.123012+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:44:23.123012+0000","last_undegraded":"2024-04-25T18:44:23.123012+0000","last_fullsized":"2024-04-25T18:44:23.123012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.420085+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:43:55.420085+0000","last_peered":"2024-04-25T18:43:55.420085+0000","last_clean":"2024-04-25T18:43:55.420085+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:43:55.420085+0000","last_undegraded":"2024-04-25T18:43:55.420085+0000","last_fullsized":"2024-04-25T18:43:55.420085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123061+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:44:23.123061+0000","last_peered":"2024-04-25T18:44:23.123061+0000","last_clean":"2024-04-25T18:44:23.123061+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:44:23.123061+0000","last_undegraded":"2024-04-25T18:44:23.123061+0000","last_fullsized":"2024-04-25T18:44:23.123061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.421826+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:43:55.421826+0000","last_peered":"2024-04-25T18:43:55.421826+0000","last_clean":"2024-04-25T18:43:55.421826+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:43:55.421826+0000","last_undegraded":"2024-04-25T18:43:55.421826+0000","last_fullsized":"2024-04-25T18:43:55.421826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"96'2","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.080026+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:44:23.080026+0000","last_peered":"2024-04-25T18:44:23.080026+0000","last_clean":"2024-04-25T18:44:23.080026+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:44:23.080026+0000","last_undegraded":"2024-04-25T18:44:23.080026+0000","last_fullsized":"2024-04-25T18:44:23.080026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.426079+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:43:55.426079+0000","last_peered":"2024-04-25T18:43:55.426079+0000","last_clean":"2024-04-25T18:43:55.426079+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:43:55.426079+0000","last_undegraded":"2024-04-25T18:43:55.426079+0000","last_fullsized":"2024-04-25T18:43:55.426079+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.114429+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:44:23.114429+0000","last_peered":"2024-04-25T18:44:23.114429+0000","last_clean":"2024-04-25T18:44:23.114429+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:44:23.114429+0000","last_undegraded":"2024-04-25T18:44:23.114429+0000","last_fullsized":"2024-04-25T18:44:23.114429+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351132+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:43:55.351132+0000","last_peered":"2024-04-25T18:43:55.351132+0000","last_clean":"2024-04-25T18:43:55.351132+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:43:55.351132+0000","last_undegraded":"2024-04-25T18:43:55.351132+0000","last_fullsized":"2024-04-25T18:43:55.351132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121885+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:44:23.121885+0000","last_peered":"2024-04-25T18:44:23.121885+0000","last_clean":"2024-04-25T18:44:23.121885+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:44:23.121885+0000","last_undegraded":"2024-04-25T18:44:23.121885+0000","last_fullsized":"2024-04-25T18:44:23.121885+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":48,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423107+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:43:55.423107+0000","last_peered":"2024-04-25T18:43:55.423107+0000","last_clean":"2024-04-25T18:43:55.423107+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:43:55.423107+0000","last_undegraded":"2024-04-25T18:43:55.423107+0000","last_fullsized":"2024-04-25T18:43:55.423107+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.116085+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:44:23.116085+0000","last_peered":"2024-04-25T18:44:23.116085+0000","last_clean":"2024-04-25T18:44:23.116085+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:44:23.116085+0000","last_undegraded":"2024-04-25T18:44:23.116085+0000","last_fullsized":"2024-04-25T18:44:23.116085+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351536+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:43:55.351536+0000","last_peered":"2024-04-25T18:43:55.351536+0000","last_clean":"2024-04-25T18:43:55.351536+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:43:55.351536+0000","last_undegraded":"2024-04-25T18:43:55.351536+0000","last_fullsized":"2024-04-25T18:43:55.351536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.118493+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:44:23.118493+0000","last_peered":"2024-04-25T18:44:23.118493+0000","last_clean":"2024-04-25T18:44:23.118493+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:44:23.118493+0000","last_undegraded":"2024-04-25T18:44:23.118493+0000","last_fullsized":"2024-04-25T18:44:23.118493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423670+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:43:55.423670+0000","last_peered":"2024-04-25T18:43:55.423670+0000","last_clean":"2024-04-25T18:43:55.423670+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:43:55.423670+0000","last_undegraded":"2024-04-25T18:43:55.423670+0000","last_fullsized":"2024-04-25T18:43:55.423670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123264+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:44:23.123264+0000","last_peered":"2024-04-25T18:44:23.123264+0000","last_clean":"2024-04-25T18:44:23.123264+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:44:23.123264+0000","last_undegraded":"2024-04-25T18:44:23.123264+0000","last_fullsized":"2024-04-25T18:44:23.123264+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.354693+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:43:55.354693+0000","last_peered":"2024-04-25T18:43:55.354693+0000","last_clean":"2024-04-25T18:43:55.354693+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:43:55.354693+0000","last_undegraded":"2024-04-25T18:43:55.354693+0000","last_fullsized":"2024-04-25T18:43:55.354693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.106205+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:44:23.106205+0000","last_peered":"2024-04-25T18:44:23.106205+0000","last_clean":"2024-04-25T18:44:23.106205+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:44:23.106205+0000","last_undegraded":"2024-04-25T18:44:23.106205+0000","last_fullsized":"2024-04-25T18:44:23.106205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.104576+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:44:23.104576+0000","last_peered":"2024-04-25T18:44:23.104576+0000","last_clean":"2024-04-25T18:44:23.104576+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:44:23.104576+0000","last_undegraded":"2024-04-25T18:44:23.104576+0000","last_fullsized":"2024-04-25T18:44:23.104576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"96'12","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101862+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:44:23.101862+0000","last_peered":"2024-04-25T18:44:23.101862+0000","last_clean":"2024-04-25T18:44:23.101862+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:44:23.101862+0000","last_undegraded":"2024-04-25T18:44:23.101862+0000","last_fullsized":"2024-04-25T18:44:23.101862+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123193+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:44:23.123193+0000","last_peered":"2024-04-25T18:44:23.123193+0000","last_clean":"2024-04-25T18:44:23.123193+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:44:23.123193+0000","last_undegraded":"2024-04-25T18:44:23.123193+0000","last_fullsized":"2024-04-25T18:44:23.123193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119015+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:44:23.119015+0000","last_peered":"2024-04-25T18:44:23.119015+0000","last_clean":"2024-04-25T18:44:23.119015+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:44:23.119015+0000","last_undegraded":"2024-04-25T18:44:23.119015+0000","last_fullsized":"2024-04-25T18:44:23.119015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"96'4","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115532+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:44:23.115532+0000","last_peered":"2024-04-25T18:44:23.115532+0000","last_clean":"2024-04-25T18:44:23.115532+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:44:23.115532+0000","last_undegraded":"2024-04-25T18:44:23.115532+0000","last_fullsized":"2024-04-25T18:44:23.115532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102917+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:44:23.102917+0000","last_peered":"2024-04-25T18:44:23.102917+0000","last_clean":"2024-04-25T18:44:23.102917+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:44:23.102917+0000","last_undegraded":"2024-04-25T18:44:23.102917+0000","last_fullsized":"2024-04-25T18:44:23.102917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"96'15","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102989+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:44:23.102989+0000","last_peered":"2024-04-25T18:44:23.102989+0000","last_clean":"2024-04-25T18:44:23.102989+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:44:23.102989+0000","last_undegraded":"2024-04-25T18:44:23.102989+0000","last_fullsized":"2024-04-25T18:44:23.102989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"96'2","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119802+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:44:23.119802+0000","last_peered":"2024-04-25T18:44:23.119802+0000","last_clean":"2024-04-25T18:44:23.119802+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:44:23.119802+0000","last_undegraded":"2024-04-25T18:44:23.119802+0000","last_fullsized":"2024-04-25T18:44:23.119802+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121619+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:44:23.121619+0000","last_peered":"2024-04-25T18:44:23.121619+0000","last_clean":"2024-04-25T18:44:23.121619+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:44:23.121619+0000","last_undegraded":"2024-04-25T18:44:23.121619+0000","last_fullsized":"2024-04-25T18:44:23.121619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"96'8","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.109732+0000","last_change":"2024-04-25T18:43:55.436949+0000","last_active":"2024-04-25T18:44:23.109732+0000","last_peered":"2024-04-25T18:44:23.109732+0000","last_clean":"2024-04-25T18:44:23.109732+0000","last_became_active":"2024-04-25T18:43:55.435757+0000","last_became_peered":"2024-04-25T18:43:55.435757+0000","last_unstale":"2024-04-25T18:44:23.109732+0000","last_undegraded":"2024-04-25T18:44:23.109732+0000","last_fullsized":"2024-04-25T18:44:23.109732+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:46.736524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.124314+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:44:23.124314+0000","last_peered":"2024-04-25T18:44:23.124314+0000","last_clean":"2024-04-25T18:44:23.124314+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:44:23.124314+0000","last_undegraded":"2024-04-25T18:44:23.124314+0000","last_fullsized":"2024-04-25T18:44:23.124314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"96'11","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.117540+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:44:23.117540+0000","last_peered":"2024-04-25T18:44:23.117540+0000","last_clean":"2024-04-25T18:44:23.117540+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:44:23.117540+0000","last_undegraded":"2024-04-25T18:44:23.117540+0000","last_fullsized":"2024-04-25T18:44:23.117540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"96'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119793+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:44:23.119793+0000","last_peered":"2024-04-25T18:44:23.119793+0000","last_clean":"2024-04-25T18:44:23.119793+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:44:23.119793+0000","last_undegraded":"2024-04-25T18:44:23.119793+0000","last_fullsized":"2024-04-25T18:44:23.119793+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64153,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":742,"num_read_kb":436,"num_write":171,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":192459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":514042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514549,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41204,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703948,"statfs":{"total":95995035648,"available":95952842752,"internally_reserved":0,"allocated":1748992,"data_stored":1165954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678075,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":514166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939714,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":513392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234055,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1662976,"data_stored":1106157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397580,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":513555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":573941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738480,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":514826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":574597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":513866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855447,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":514636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659220,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1675264,"data_stored":1105309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":44,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:44:56.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:44:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:56 smithi114 ceph-mon[32988]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:56 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:56 smithi114 ceph-mon[32988]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:44:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:56 smithi063 ceph-mon[28006]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:56 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:56 smithi063 ceph-mon[28006]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:44:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:56 smithi070 ceph-mon[32820]: pgmap v417: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:44:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:56 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:57.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:56 smithi070 ceph-mon[32820]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:44:57.284 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:44:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:58.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:57 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:44:58.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:57 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:44:58.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:57 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:44:59.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:44:58 smithi114 ceph-mon[32988]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:59.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:44:58 smithi063 ceph-mon[28006]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:59.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:44:58 smithi070 ceph-mon[32820]: pgmap v418: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:44:59.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.703+0000 7f155fa00700 1 -- 172.21.15.63:0/3684443141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 msgr2=0x7f155806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:59.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.703+0000 7f155fa00700 1 --2- 172.21.15.63:0/3684443141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f155806abb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1544009990 tx=0x7f1544009ca0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 -- 172.21.15.63:0/3684443141 shutdown_connections 2024-04-25T18:44:59.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 --2- 172.21.15.63:0/3684443141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f155810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 --2- 172.21.15.63:0/3684443141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 0x7f1558108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 --2- 172.21.15.63:0/3684443141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f155806abb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 -- 172.21.15.63:0/3684443141 >> 172.21.15.63:0/3684443141 conn(0x7f15580788f0 msgr2=0x7f1558078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:59.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 -- 172.21.15.63:0/3684443141 shutdown_connections 2024-04-25T18:44:59.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.704+0000 7f155fa00700 1 -- 172.21.15.63:0/3684443141 wait complete. 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.705+0000 7f155fa00700 1 Processor -- start 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.705+0000 7f155fa00700 1 -- start start 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f1558199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 0x7f155819da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1558112c20 con 0x7f155806b0f0 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1558112aa0 con 0x7f1558108a30 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155fa00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1558112da0 con 0x7f155806a750 2024-04-25T18:44:59.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:59.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f1557e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f1558199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:59.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41688/0 (socket says 172.21.15.63:41688) 2024-04-25T18:44:59.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155d000700 1 -- 172.21.15.63:0/3382072182 learned_addr learned my addr 172.21.15.63:0/3382072182 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:44:59.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.706+0000 7f155da00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 0x7f155819da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 -- 172.21.15.63:0/3382072182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 msgr2=0x7f1558199210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 -- 172.21.15.63:0/3382072182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 msgr2=0x7f155819da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 0x7f155819da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 -- 172.21.15.63:0/3382072182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155819e1d0 con 0x7f155806b0f0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f155d000700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1557e00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f1558199750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f15440098e0 tx=0x7f15440057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.707+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544027070 con 0x7f155806b0f0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.708+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1544022470 con 0x7f155806b0f0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.708+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155819e460 con 0x7f155806b0f0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.708+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544017650 con 0x7f155806b0f0 2024-04-25T18:44:59.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.708+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1558101940 con 0x7f155806b0f0 2024-04-25T18:44:59.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.709+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1558104170 con 0x7f155806b0f0 2024-04-25T18:44:59.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.713+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f154402a070 con 0x7f155806b0f0 2024-04-25T18:44:59.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.714+0000 7f1555600700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 0x7f15400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:44:59.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.714+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f15440d13e0 con 0x7f155806b0f0 2024-04-25T18:44:59.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.714+0000 7f155d000700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 0x7f15400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:44:59.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.715+0000 7f155d000700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 0x7f15400a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f154c011550 tx=0x7f154c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:44:59.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.717+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1544031080 con 0x7f155806b0f0 2024-04-25T18:44:59.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.943+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1558069730 con 0x7f15400a3c40 2024-04-25T18:44:59.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.952+0000 7f1555600700 1 -- 172.21.15.63:0/3382072182 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258850 (secure 0 0 0) 0x7f1558069730 con 0x7f15400a3c40 2024-04-25T18:44:59.955 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.957+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 msgr2=0x7f15400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.957+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 0x7f15400a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f154c011550 tx=0x7f154c009260 comp rx=0 tx=0).stop 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.957+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 msgr2=0x7f1558199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.957+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f1558199750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f15440098e0 tx=0x7f15440057c0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 shutdown_connections 2024-04-25T18:44:59.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f15400a3c40 0x7f15400a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f155806a750 0x7f1558199210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1558108a30 0x7f155819da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 --2- 172.21.15.63:0/3382072182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f155806b0f0 0x7f1558199750 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 >> 172.21.15.63:0/3382072182 conn(0x7f15580788f0 msgr2=0x7f15580ffa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 shutdown_connections 2024-04-25T18:44:59.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:44:59.958+0000 7f155fa00700 1 -- 172.21.15.63:0/3382072182 wait complete. 2024-04-25T18:44:59.962 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:45:00.844 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-04-25T18:44:59.188410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654755,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":908,"num_read_kb":570,"num_write":354,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":392732,"kb_used_data":14460,"kb_used_omap":18,"kb_used_meta":378157,"kb_avail":1124549092,"statfs":{"total":1151940427776,"available":1151538270208,"internally_reserved":0,"allocated":14807040,"data_stored":8124441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":387233081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005076"},"pg_stats":[{"pgid":"3.3f","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115736+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:44:23.115736+0000","last_peered":"2024-04-25T18:44:23.115736+0000","last_clean":"2024-04-25T18:44:23.115736+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:44:23.115736+0000","last_undegraded":"2024-04-25T18:44:23.115736+0000","last_fullsized":"2024-04-25T18:44:23.115736+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"96'4","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.112259+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:44:23.112259+0000","last_peered":"2024-04-25T18:44:23.112259+0000","last_clean":"2024-04-25T18:44:23.112259+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:44:23.112259+0000","last_undegraded":"2024-04-25T18:44:23.112259+0000","last_fullsized":"2024-04-25T18:44:23.112259+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.117310+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:44:23.117310+0000","last_peered":"2024-04-25T18:44:23.117310+0000","last_clean":"2024-04-25T18:44:23.117310+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:44:23.117310+0000","last_undegraded":"2024-04-25T18:44:23.117310+0000","last_fullsized":"2024-04-25T18:44:23.117310+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.098636+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:44:23.098636+0000","last_peered":"2024-04-25T18:44:23.098636+0000","last_clean":"2024-04-25T18:44:23.098636+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:44:23.098636+0000","last_undegraded":"2024-04-25T18:44:23.098636+0000","last_fullsized":"2024-04-25T18:44:23.098636+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"96'4","reported_seq":28,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101893+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:44:23.101893+0000","last_peered":"2024-04-25T18:44:23.101893+0000","last_clean":"2024-04-25T18:44:23.101893+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:44:23.101893+0000","last_undegraded":"2024-04-25T18:44:23.101893+0000","last_fullsized":"2024-04-25T18:44:23.101893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"96'6","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102130+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:44:23.102130+0000","last_peered":"2024-04-25T18:44:23.102130+0000","last_clean":"2024-04-25T18:44:23.102130+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:44:23.102130+0000","last_undegraded":"2024-04-25T18:44:23.102130+0000","last_fullsized":"2024-04-25T18:44:23.102130+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"96'5","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.106962+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:44:23.106962+0000","last_peered":"2024-04-25T18:44:23.106962+0000","last_clean":"2024-04-25T18:44:23.106962+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:44:23.106962+0000","last_undegraded":"2024-04-25T18:44:23.106962+0000","last_fullsized":"2024-04-25T18:44:23.106962+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.122311+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:44:23.122311+0000","last_peered":"2024-04-25T18:44:23.122311+0000","last_clean":"2024-04-25T18:44:23.122311+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:44:23.122311+0000","last_undegraded":"2024-04-25T18:44:23.122311+0000","last_fullsized":"2024-04-25T18:44:23.122311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'11","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.117685+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:44:23.117685+0000","last_peered":"2024-04-25T18:44:23.117685+0000","last_clean":"2024-04-25T18:44:23.117685+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:44:23.117685+0000","last_undegraded":"2024-04-25T18:44:23.117685+0000","last_fullsized":"2024-04-25T18:44:23.117685+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"96'16","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103556+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:44:23.103556+0000","last_peered":"2024-04-25T18:44:23.103556+0000","last_clean":"2024-04-25T18:44:23.103556+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:44:23.103556+0000","last_undegraded":"2024-04-25T18:44:23.103556+0000","last_fullsized":"2024-04-25T18:44:23.103556+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115432+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:44:23.115432+0000","last_peered":"2024-04-25T18:44:23.115432+0000","last_clean":"2024-04-25T18:44:23.115432+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:44:23.115432+0000","last_undegraded":"2024-04-25T18:44:23.115432+0000","last_fullsized":"2024-04-25T18:44:23.115432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.093830+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:44:23.093830+0000","last_peered":"2024-04-25T18:44:23.093830+0000","last_clean":"2024-04-25T18:44:23.093830+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:44:23.093830+0000","last_undegraded":"2024-04-25T18:44:23.093830+0000","last_fullsized":"2024-04-25T18:44:23.093830+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.108614+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:44:23.108614+0000","last_peered":"2024-04-25T18:44:23.108614+0000","last_clean":"2024-04-25T18:44:23.108614+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:44:23.108614+0000","last_undegraded":"2024-04-25T18:44:23.108614+0000","last_fullsized":"2024-04-25T18:44:23.108614+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.113639+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:44:23.113639+0000","last_peered":"2024-04-25T18:44:23.113639+0000","last_clean":"2024-04-25T18:44:23.113639+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:44:23.113639+0000","last_undegraded":"2024-04-25T18:44:23.113639+0000","last_fullsized":"2024-04-25T18:44:23.113639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115349+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:44:23.115349+0000","last_peered":"2024-04-25T18:44:23.115349+0000","last_clean":"2024-04-25T18:44:23.115349+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:44:23.115349+0000","last_undegraded":"2024-04-25T18:44:23.115349+0000","last_fullsized":"2024-04-25T18:44:23.115349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119296+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:44:23.119296+0000","last_peered":"2024-04-25T18:44:23.119296+0000","last_clean":"2024-04-25T18:44:23.119296+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:44:23.119296+0000","last_undegraded":"2024-04-25T18:44:23.119296+0000","last_fullsized":"2024-04-25T18:44:23.119296+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115732+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:44:23.115732+0000","last_peered":"2024-04-25T18:44:23.115732+0000","last_clean":"2024-04-25T18:44:23.115732+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:44:23.115732+0000","last_undegraded":"2024-04-25T18:44:23.115732+0000","last_fullsized":"2024-04-25T18:44:23.115732+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121838+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:44:23.121838+0000","last_peered":"2024-04-25T18:44:23.121838+0000","last_clean":"2024-04-25T18:44:23.121838+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:44:23.121838+0000","last_undegraded":"2024-04-25T18:44:23.121838+0000","last_fullsized":"2024-04-25T18:44:23.121838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.120929+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:44:23.120929+0000","last_peered":"2024-04-25T18:44:23.120929+0000","last_clean":"2024-04-25T18:44:23.120929+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:44:23.120929+0000","last_undegraded":"2024-04-25T18:44:23.120929+0000","last_fullsized":"2024-04-25T18:44:23.120929+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.359876+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:43:55.359876+0000","last_peered":"2024-04-25T18:43:55.359876+0000","last_clean":"2024-04-25T18:43:55.359876+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:43:55.359876+0000","last_undegraded":"2024-04-25T18:43:55.359876+0000","last_fullsized":"2024-04-25T18:43:55.359876+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119893+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:44:23.119893+0000","last_peered":"2024-04-25T18:44:23.119893+0000","last_clean":"2024-04-25T18:44:23.119893+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:44:23.119893+0000","last_undegraded":"2024-04-25T18:44:23.119893+0000","last_fullsized":"2024-04-25T18:44:23.119893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.428032+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:43:55.428032+0000","last_peered":"2024-04-25T18:43:55.428032+0000","last_clean":"2024-04-25T18:43:55.428032+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:43:55.428032+0000","last_undegraded":"2024-04-25T18:43:55.428032+0000","last_fullsized":"2024-04-25T18:43:55.428032+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121560+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:44:23.121560+0000","last_peered":"2024-04-25T18:44:23.121560+0000","last_clean":"2024-04-25T18:44:23.121560+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:44:23.121560+0000","last_undegraded":"2024-04-25T18:44:23.121560+0000","last_fullsized":"2024-04-25T18:44:23.121560+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423781+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:43:55.423781+0000","last_peered":"2024-04-25T18:43:55.423781+0000","last_clean":"2024-04-25T18:43:55.423781+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:43:55.423781+0000","last_undegraded":"2024-04-25T18:43:55.423781+0000","last_fullsized":"2024-04-25T18:43:55.423781+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"96'9","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123522+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:44:23.123522+0000","last_peered":"2024-04-25T18:44:23.123522+0000","last_clean":"2024-04-25T18:44:23.123522+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:44:23.123522+0000","last_undegraded":"2024-04-25T18:44:23.123522+0000","last_fullsized":"2024-04-25T18:44:23.123522+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.424006+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:43:55.424006+0000","last_peered":"2024-04-25T18:43:55.424006+0000","last_clean":"2024-04-25T18:43:55.424006+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:43:55.424006+0000","last_undegraded":"2024-04-25T18:43:55.424006+0000","last_fullsized":"2024-04-25T18:43:55.424006+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"96'15","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.105716+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:44:23.105716+0000","last_peered":"2024-04-25T18:44:23.105716+0000","last_clean":"2024-04-25T18:44:23.105716+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:44:23.105716+0000","last_undegraded":"2024-04-25T18:44:23.105716+0000","last_fullsized":"2024-04-25T18:44:23.105716+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"92'5","reported_seq":163,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:57.804760+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:44:57.804760+0000","last_peered":"2024-04-25T18:44:57.804760+0000","last_clean":"2024-04-25T18:44:57.804760+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:44:57.804760+0000","last_undegraded":"2024-04-25T18:44:57.804760+0000","last_fullsized":"2024-04-25T18:44:57.804760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":127,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"96'7","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103869+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:44:23.103869+0000","last_peered":"2024-04-25T18:44:23.103869+0000","last_clean":"2024-04-25T18:44:23.103869+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:44:23.103869+0000","last_undegraded":"2024-04-25T18:44:23.103869+0000","last_fullsized":"2024-04-25T18:44:23.103869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.350552+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:43:55.350552+0000","last_peered":"2024-04-25T18:43:55.350552+0000","last_clean":"2024-04-25T18:43:55.350552+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:43:55.350552+0000","last_undegraded":"2024-04-25T18:43:55.350552+0000","last_fullsized":"2024-04-25T18:43:55.350552+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121444+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:44:23.121444+0000","last_peered":"2024-04-25T18:44:23.121444+0000","last_clean":"2024-04-25T18:44:23.121444+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:44:23.121444+0000","last_undegraded":"2024-04-25T18:44:23.121444+0000","last_fullsized":"2024-04-25T18:44:23.121444+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.354051+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:43:55.354051+0000","last_peered":"2024-04-25T18:43:55.354051+0000","last_clean":"2024-04-25T18:43:55.354051+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:43:55.354051+0000","last_undegraded":"2024-04-25T18:43:55.354051+0000","last_fullsized":"2024-04-25T18:43:55.354051+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123009+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:44:23.123009+0000","last_peered":"2024-04-25T18:44:23.123009+0000","last_clean":"2024-04-25T18:44:23.123009+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:44:23.123009+0000","last_undegraded":"2024-04-25T18:44:23.123009+0000","last_fullsized":"2024-04-25T18:44:23.123009+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423469+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:43:55.423469+0000","last_peered":"2024-04-25T18:43:55.423469+0000","last_clean":"2024-04-25T18:43:55.423469+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:43:55.423469+0000","last_undegraded":"2024-04-25T18:43:55.423469+0000","last_fullsized":"2024-04-25T18:43:55.423469+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'11","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.122151+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:44:23.122151+0000","last_peered":"2024-04-25T18:44:23.122151+0000","last_clean":"2024-04-25T18:44:23.122151+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:44:23.122151+0000","last_undegraded":"2024-04-25T18:44:23.122151+0000","last_fullsized":"2024-04-25T18:44:23.122151+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"92'3","reported_seq":158,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:55.801087+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:44:55.801087+0000","last_peered":"2024-04-25T18:44:55.801087+0000","last_clean":"2024-04-25T18:44:55.801087+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:44:55.801087+0000","last_undegraded":"2024-04-25T18:44:55.801087+0000","last_fullsized":"2024-04-25T18:44:55.801087+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.122809+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:44:23.122809+0000","last_peered":"2024-04-25T18:44:23.122809+0000","last_clean":"2024-04-25T18:44:23.122809+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:44:23.122809+0000","last_undegraded":"2024-04-25T18:44:23.122809+0000","last_fullsized":"2024-04-25T18:44:23.122809+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.434184+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:43:55.434184+0000","last_peered":"2024-04-25T18:43:55.434184+0000","last_clean":"2024-04-25T18:43:55.434184+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:43:55.434184+0000","last_undegraded":"2024-04-25T18:43:55.434184+0000","last_fullsized":"2024-04-25T18:43:55.434184+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.118604+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:44:23.118604+0000","last_peered":"2024-04-25T18:44:23.118604+0000","last_clean":"2024-04-25T18:44:23.118604+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:44:23.118604+0000","last_undegraded":"2024-04-25T18:44:23.118604+0000","last_fullsized":"2024-04-25T18:44:23.118604+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423919+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:43:55.423919+0000","last_peered":"2024-04-25T18:43:55.423919+0000","last_clean":"2024-04-25T18:43:55.423919+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:43:55.423919+0000","last_undegraded":"2024-04-25T18:43:55.423919+0000","last_fullsized":"2024-04-25T18:43:55.423919+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"96'11","reported_seq":42,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119652+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:44:23.119652+0000","last_peered":"2024-04-25T18:44:23.119652+0000","last_clean":"2024-04-25T18:44:23.119652+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:44:23.119652+0000","last_undegraded":"2024-04-25T18:44:23.119652+0000","last_fullsized":"2024-04-25T18:44:23.119652+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"96'32","reported_seq":321,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:56.802804+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:44:56.802804+0000","last_peered":"2024-04-25T18:44:56.802804+0000","last_clean":"2024-04-25T18:44:56.802804+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:44:56.802804+0000","last_undegraded":"2024-04-25T18:44:56.802804+0000","last_fullsized":"2024-04-25T18:44:56.802804+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60011,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":258,"num_read_kb":187,"num_write":33,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.099067+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:44:23.099067+0000","last_peered":"2024-04-25T18:44:23.099067+0000","last_clean":"2024-04-25T18:44:23.099067+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:44:23.099067+0000","last_undegraded":"2024-04-25T18:44:23.099067+0000","last_fullsized":"2024-04-25T18:44:23.099067+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":38,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351253+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:43:55.351253+0000","last_peered":"2024-04-25T18:43:55.351253+0000","last_clean":"2024-04-25T18:43:55.351253+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:43:55.351253+0000","last_undegraded":"2024-04-25T18:43:55.351253+0000","last_fullsized":"2024-04-25T18:43:55.351253+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.124289+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:44:23.124289+0000","last_peered":"2024-04-25T18:44:23.124289+0000","last_clean":"2024-04-25T18:44:23.124289+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:44:23.124289+0000","last_undegraded":"2024-04-25T18:44:23.124289+0000","last_fullsized":"2024-04-25T18:44:23.124289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423445+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:43:55.423445+0000","last_peered":"2024-04-25T18:43:55.423445+0000","last_clean":"2024-04-25T18:43:55.423445+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:43:55.423445+0000","last_undegraded":"2024-04-25T18:43:55.423445+0000","last_fullsized":"2024-04-25T18:43:55.423445+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'15","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.118162+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:44:23.118162+0000","last_peered":"2024-04-25T18:44:23.118162+0000","last_clean":"2024-04-25T18:44:23.118162+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:44:23.118162+0000","last_undegraded":"2024-04-25T18:44:23.118162+0000","last_fullsized":"2024-04-25T18:44:23.118162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"96'5","reported_seq":165,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:56.881168+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:44:56.881168+0000","last_peered":"2024-04-25T18:44:56.881168+0000","last_clean":"2024-04-25T18:44:56.881168+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:44:56.881168+0000","last_undegraded":"2024-04-25T18:44:56.881168+0000","last_fullsized":"2024-04-25T18:44:56.881168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101174+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:44:23.101174+0000","last_peered":"2024-04-25T18:44:23.101174+0000","last_clean":"2024-04-25T18:44:23.101174+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:44:23.101174+0000","last_undegraded":"2024-04-25T18:44:23.101174+0000","last_fullsized":"2024-04-25T18:44:23.101174+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":41,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.434037+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:43:55.434037+0000","last_peered":"2024-04-25T18:43:55.434037+0000","last_clean":"2024-04-25T18:43:55.434037+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:43:55.434037+0000","last_undegraded":"2024-04-25T18:43:55.434037+0000","last_fullsized":"2024-04-25T18:43:55.434037+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.105260+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:44:23.105260+0000","last_peered":"2024-04-25T18:44:23.105260+0000","last_clean":"2024-04-25T18:44:23.105260+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:44:23.105260+0000","last_undegraded":"2024-04-25T18:44:23.105260+0000","last_fullsized":"2024-04-25T18:44:23.105260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"92'2","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423704+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:43:55.423704+0000","last_peered":"2024-04-25T18:43:55.423704+0000","last_clean":"2024-04-25T18:43:55.423704+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:43:55.423704+0000","last_undegraded":"2024-04-25T18:43:55.423704+0000","last_fullsized":"2024-04-25T18:43:55.423704+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.103933+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:44:23.103933+0000","last_peered":"2024-04-25T18:44:23.103933+0000","last_clean":"2024-04-25T18:44:23.103933+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:44:23.103933+0000","last_undegraded":"2024-04-25T18:44:23.103933+0000","last_fullsized":"2024-04-25T18:44:23.103933+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.359384+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:43:55.359384+0000","last_peered":"2024-04-25T18:43:55.359384+0000","last_clean":"2024-04-25T18:43:55.359384+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:43:55.359384+0000","last_undegraded":"2024-04-25T18:43:55.359384+0000","last_fullsized":"2024-04-25T18:43:55.359384+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123292+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:44:23.123292+0000","last_peered":"2024-04-25T18:44:23.123292+0000","last_clean":"2024-04-25T18:44:23.123292+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:44:23.123292+0000","last_undegraded":"2024-04-25T18:44:23.123292+0000","last_fullsized":"2024-04-25T18:44:23.123292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423474+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:43:55.423474+0000","last_peered":"2024-04-25T18:43:55.423474+0000","last_clean":"2024-04-25T18:43:55.423474+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:43:55.423474+0000","last_undegraded":"2024-04-25T18:43:55.423474+0000","last_fullsized":"2024-04-25T18:43:55.423474+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.114766+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:44:23.114766+0000","last_peered":"2024-04-25T18:44:23.114766+0000","last_clean":"2024-04-25T18:44:23.114766+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:44:23.114766+0000","last_undegraded":"2024-04-25T18:44:23.114766+0000","last_fullsized":"2024-04-25T18:44:23.114766+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"96'8","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115302+0000","last_change":"2024-04-25T18:43:55.423277+0000","last_active":"2024-04-25T18:44:23.115302+0000","last_peered":"2024-04-25T18:44:23.115302+0000","last_clean":"2024-04-25T18:44:23.115302+0000","last_became_active":"2024-04-25T18:43:55.421960+0000","last_became_peered":"2024-04-25T18:43:55.421960+0000","last_unstale":"2024-04-25T18:44:23.115302+0000","last_undegraded":"2024-04-25T18:44:23.115302+0000","last_fullsized":"2024-04-25T18:44:23.115302+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:27.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"96'5","reported_seq":160,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:55.878825+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:44:55.878825+0000","last_peered":"2024-04-25T18:44:55.878825+0000","last_clean":"2024-04-25T18:44:55.878825+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:44:55.878825+0000","last_undegraded":"2024-04-25T18:44:55.878825+0000","last_fullsized":"2024-04-25T18:44:55.878825+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123069+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:44:23.123069+0000","last_peered":"2024-04-25T18:44:23.123069+0000","last_clean":"2024-04-25T18:44:23.123069+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:44:23.123069+0000","last_undegraded":"2024-04-25T18:44:23.123069+0000","last_fullsized":"2024-04-25T18:44:23.123069+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.420012+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:43:55.420012+0000","last_peered":"2024-04-25T18:43:55.420012+0000","last_clean":"2024-04-25T18:43:55.420012+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:43:55.420012+0000","last_undegraded":"2024-04-25T18:43:55.420012+0000","last_fullsized":"2024-04-25T18:43:55.420012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":366,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423348+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:43:55.423348+0000","last_peered":"2024-04-25T18:43:55.423348+0000","last_clean":"2024-04-25T18:43:55.423348+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:43:55.423348+0000","last_undegraded":"2024-04-25T18:43:55.423348+0000","last_fullsized":"2024-04-25T18:43:55.423348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102301+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:44:23.102301+0000","last_peered":"2024-04-25T18:44:23.102301+0000","last_clean":"2024-04-25T18:44:23.102301+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:44:23.102301+0000","last_undegraded":"2024-04-25T18:44:23.102301+0000","last_fullsized":"2024-04-25T18:44:23.102301+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423588+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:43:55.423588+0000","last_peered":"2024-04-25T18:43:55.423588+0000","last_clean":"2024-04-25T18:43:55.423588+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:43:55.423588+0000","last_undegraded":"2024-04-25T18:43:55.423588+0000","last_fullsized":"2024-04-25T18:43:55.423588+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423631+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:43:55.423631+0000","last_peered":"2024-04-25T18:43:55.423631+0000","last_clean":"2024-04-25T18:43:55.423631+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:43:55.423631+0000","last_undegraded":"2024-04-25T18:43:55.423631+0000","last_fullsized":"2024-04-25T18:43:55.423631+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.112759+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:44:23.112759+0000","last_peered":"2024-04-25T18:44:23.112759+0000","last_clean":"2024-04-25T18:44:23.112759+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:44:23.112759+0000","last_undegraded":"2024-04-25T18:44:23.112759+0000","last_fullsized":"2024-04-25T18:44:23.112759+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"96'12","reported_seq":43,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.918564+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:44:23.918564+0000","last_peered":"2024-04-25T18:44:23.918564+0000","last_clean":"2024-04-25T18:44:23.918564+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:44:23.918564+0000","last_undegraded":"2024-04-25T18:44:23.918564+0000","last_fullsized":"2024-04-25T18:44:23.918564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123012+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:44:23.123012+0000","last_peered":"2024-04-25T18:44:23.123012+0000","last_clean":"2024-04-25T18:44:23.123012+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:44:23.123012+0000","last_undegraded":"2024-04-25T18:44:23.123012+0000","last_fullsized":"2024-04-25T18:44:23.123012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.420085+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:43:55.420085+0000","last_peered":"2024-04-25T18:43:55.420085+0000","last_clean":"2024-04-25T18:43:55.420085+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:43:55.420085+0000","last_undegraded":"2024-04-25T18:43:55.420085+0000","last_fullsized":"2024-04-25T18:43:55.420085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123061+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:44:23.123061+0000","last_peered":"2024-04-25T18:44:23.123061+0000","last_clean":"2024-04-25T18:44:23.123061+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:44:23.123061+0000","last_undegraded":"2024-04-25T18:44:23.123061+0000","last_fullsized":"2024-04-25T18:44:23.123061+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.421826+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:43:55.421826+0000","last_peered":"2024-04-25T18:43:55.421826+0000","last_clean":"2024-04-25T18:43:55.421826+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:43:55.421826+0000","last_undegraded":"2024-04-25T18:43:55.421826+0000","last_fullsized":"2024-04-25T18:43:55.421826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"96'2","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.080026+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:44:23.080026+0000","last_peered":"2024-04-25T18:44:23.080026+0000","last_clean":"2024-04-25T18:44:23.080026+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:44:23.080026+0000","last_undegraded":"2024-04-25T18:44:23.080026+0000","last_fullsized":"2024-04-25T18:44:23.080026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.426079+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:43:55.426079+0000","last_peered":"2024-04-25T18:43:55.426079+0000","last_clean":"2024-04-25T18:43:55.426079+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:43:55.426079+0000","last_undegraded":"2024-04-25T18:43:55.426079+0000","last_fullsized":"2024-04-25T18:43:55.426079+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.114429+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:44:23.114429+0000","last_peered":"2024-04-25T18:44:23.114429+0000","last_clean":"2024-04-25T18:44:23.114429+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:44:23.114429+0000","last_undegraded":"2024-04-25T18:44:23.114429+0000","last_fullsized":"2024-04-25T18:44:23.114429+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351132+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:43:55.351132+0000","last_peered":"2024-04-25T18:43:55.351132+0000","last_clean":"2024-04-25T18:43:55.351132+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:43:55.351132+0000","last_undegraded":"2024-04-25T18:43:55.351132+0000","last_fullsized":"2024-04-25T18:43:55.351132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121885+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:44:23.121885+0000","last_peered":"2024-04-25T18:44:23.121885+0000","last_clean":"2024-04-25T18:44:23.121885+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:44:23.121885+0000","last_undegraded":"2024-04-25T18:44:23.121885+0000","last_fullsized":"2024-04-25T18:44:23.121885+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":48,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423107+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:43:55.423107+0000","last_peered":"2024-04-25T18:43:55.423107+0000","last_clean":"2024-04-25T18:43:55.423107+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:43:55.423107+0000","last_undegraded":"2024-04-25T18:43:55.423107+0000","last_fullsized":"2024-04-25T18:43:55.423107+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"96'5","reported_seq":30,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.116085+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:44:23.116085+0000","last_peered":"2024-04-25T18:44:23.116085+0000","last_clean":"2024-04-25T18:44:23.116085+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:44:23.116085+0000","last_undegraded":"2024-04-25T18:44:23.116085+0000","last_fullsized":"2024-04-25T18:44:23.116085+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.351536+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:43:55.351536+0000","last_peered":"2024-04-25T18:43:55.351536+0000","last_clean":"2024-04-25T18:43:55.351536+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:43:55.351536+0000","last_undegraded":"2024-04-25T18:43:55.351536+0000","last_fullsized":"2024-04-25T18:43:55.351536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.118493+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:44:23.118493+0000","last_peered":"2024-04-25T18:44:23.118493+0000","last_clean":"2024-04-25T18:44:23.118493+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:44:23.118493+0000","last_undegraded":"2024-04-25T18:44:23.118493+0000","last_fullsized":"2024-04-25T18:44:23.118493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"92'1","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.423670+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:43:55.423670+0000","last_peered":"2024-04-25T18:43:55.423670+0000","last_clean":"2024-04-25T18:43:55.423670+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:43:55.423670+0000","last_undegraded":"2024-04-25T18:43:55.423670+0000","last_fullsized":"2024-04-25T18:43:55.423670+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123264+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:44:23.123264+0000","last_peered":"2024-04-25T18:44:23.123264+0000","last_clean":"2024-04-25T18:44:23.123264+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:44:23.123264+0000","last_undegraded":"2024-04-25T18:44:23.123264+0000","last_fullsized":"2024-04-25T18:44:23.123264+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:43:55.354693+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:43:55.354693+0000","last_peered":"2024-04-25T18:43:55.354693+0000","last_clean":"2024-04-25T18:43:55.354693+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:43:55.354693+0000","last_undegraded":"2024-04-25T18:43:55.354693+0000","last_fullsized":"2024-04-25T18:43:55.354693+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.106205+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:44:23.106205+0000","last_peered":"2024-04-25T18:44:23.106205+0000","last_clean":"2024-04-25T18:44:23.106205+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:44:23.106205+0000","last_undegraded":"2024-04-25T18:44:23.106205+0000","last_fullsized":"2024-04-25T18:44:23.106205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"96'9","reported_seq":34,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.104576+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:44:23.104576+0000","last_peered":"2024-04-25T18:44:23.104576+0000","last_clean":"2024-04-25T18:44:23.104576+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:44:23.104576+0000","last_undegraded":"2024-04-25T18:44:23.104576+0000","last_fullsized":"2024-04-25T18:44:23.104576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"96'12","reported_seq":37,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.101862+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:44:23.101862+0000","last_peered":"2024-04-25T18:44:23.101862+0000","last_clean":"2024-04-25T18:44:23.101862+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:44:23.101862+0000","last_undegraded":"2024-04-25T18:44:23.101862+0000","last_fullsized":"2024-04-25T18:44:23.101862+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"96'8","reported_seq":33,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.123193+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:44:23.123193+0000","last_peered":"2024-04-25T18:44:23.123193+0000","last_clean":"2024-04-25T18:44:23.123193+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:44:23.123193+0000","last_undegraded":"2024-04-25T18:44:23.123193+0000","last_fullsized":"2024-04-25T18:44:23.123193+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"96'7","reported_seq":32,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119015+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:44:23.119015+0000","last_peered":"2024-04-25T18:44:23.119015+0000","last_clean":"2024-04-25T18:44:23.119015+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:44:23.119015+0000","last_undegraded":"2024-04-25T18:44:23.119015+0000","last_fullsized":"2024-04-25T18:44:23.119015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"96'4","reported_seq":29,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.115532+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:44:23.115532+0000","last_peered":"2024-04-25T18:44:23.115532+0000","last_clean":"2024-04-25T18:44:23.115532+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:44:23.115532+0000","last_undegraded":"2024-04-25T18:44:23.115532+0000","last_fullsized":"2024-04-25T18:44:23.115532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102917+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:44:23.102917+0000","last_peered":"2024-04-25T18:44:23.102917+0000","last_clean":"2024-04-25T18:44:23.102917+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:44:23.102917+0000","last_undegraded":"2024-04-25T18:44:23.102917+0000","last_fullsized":"2024-04-25T18:44:23.102917+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"96'15","reported_seq":40,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.102989+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:44:23.102989+0000","last_peered":"2024-04-25T18:44:23.102989+0000","last_clean":"2024-04-25T18:44:23.102989+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:44:23.102989+0000","last_undegraded":"2024-04-25T18:44:23.102989+0000","last_fullsized":"2024-04-25T18:44:23.102989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"96'2","reported_seq":27,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119802+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:44:23.119802+0000","last_peered":"2024-04-25T18:44:23.119802+0000","last_clean":"2024-04-25T18:44:23.119802+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:44:23.119802+0000","last_undegraded":"2024-04-25T18:44:23.119802+0000","last_fullsized":"2024-04-25T18:44:23.119802+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'6","reported_seq":31,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.121619+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:44:23.121619+0000","last_peered":"2024-04-25T18:44:23.121619+0000","last_clean":"2024-04-25T18:44:23.121619+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:44:23.121619+0000","last_undegraded":"2024-04-25T18:44:23.121619+0000","last_fullsized":"2024-04-25T18:44:23.121619+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"96'8","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.109732+0000","last_change":"2024-04-25T18:43:55.436949+0000","last_active":"2024-04-25T18:44:23.109732+0000","last_peered":"2024-04-25T18:44:23.109732+0000","last_clean":"2024-04-25T18:44:23.109732+0000","last_became_active":"2024-04-25T18:43:55.435757+0000","last_became_peered":"2024-04-25T18:43:55.435757+0000","last_unstale":"2024-04-25T18:44:23.109732+0000","last_undegraded":"2024-04-25T18:44:23.109732+0000","last_fullsized":"2024-04-25T18:44:23.109732+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:46.736524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"96'10","reported_seq":35,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.124314+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:44:23.124314+0000","last_peered":"2024-04-25T18:44:23.124314+0000","last_clean":"2024-04-25T18:44:23.124314+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:44:23.124314+0000","last_undegraded":"2024-04-25T18:44:23.124314+0000","last_fullsized":"2024-04-25T18:44:23.124314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"96'11","reported_seq":36,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.117540+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:44:23.117540+0000","last_peered":"2024-04-25T18:44:23.117540+0000","last_clean":"2024-04-25T18:44:23.117540+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:44:23.117540+0000","last_undegraded":"2024-04-25T18:44:23.117540+0000","last_fullsized":"2024-04-25T18:44:23.117540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"96'8","reported_seq":39,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-25T18:44:23.119793+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:44:23.119793+0000","last_peered":"2024-04-25T18:44:23.119793+0000","last_clean":"2024-04-25T18:44:23.119793+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:44:23.119793+0000","last_undegraded":"2024-04-25T18:44:23.119793+0000","last_fullsized":"2024-04-25T18:44:23.119793+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64153,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":772,"num_read_kb":451,"num_write":171,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":192459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252913,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":514042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514550,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41204,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703948,"statfs":{"total":95995035648,"available":95952842752,"internally_reserved":0,"allocated":1748992,"data_stored":1165954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678076,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1069056,"data_stored":514166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939715,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":513392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234055,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1662976,"data_stored":1106157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397581,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":513555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":573941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738481,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":514826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":574597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117085,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":513866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":514636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659221,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32812,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712340,"statfs":{"total":95995035648,"available":95961436160,"internally_reserved":0,"allocated":1675264,"data_stored":1105309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":61249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":44,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:45:00.848 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T18:45:00.848 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T18:45:00.848 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T18:45:00.848 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T18:45:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:00 smithi114 ceph-mon[32988]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:00 smithi114 ceph-mon[32988]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:45:01.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:00 smithi063 ceph-mon[28006]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:01.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:00 smithi063 ceph-mon[28006]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:45:01.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:00 smithi070 ceph-mon[32820]: pgmap v419: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:01.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:00 smithi070 ceph-mon[32820]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:45:01.343 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:02 smithi114 ceph-mon[32988]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:03.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:02 smithi063 ceph-mon[28006]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:03.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:02 smithi070 ceph-mon[32820]: pgmap v420: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:03.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.671+0000 7f861e000700 1 -- 172.21.15.63:0/1539279586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806a750 msgr2=0x7f861806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:03.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.671+0000 7f861e000700 1 --2- 172.21.15.63:0/1539279586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806a750 0x7f861806ab30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8608009f90 tx=0x7f860800b330 comp rx=0 tx=0).stop 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 -- 172.21.15.63:0/1539279586 shutdown_connections 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 --2- 172.21.15.63:0/1539279586 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806a750 0x7f861806ab30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 --2- 172.21.15.63:0/1539279586 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f86181118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 --2- 172.21.15.63:0/1539279586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806b070 0x7f861810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 -- 172.21.15.63:0/1539279586 >> 172.21.15.63:0/1539279586 conn(0x7f86180788f0 msgr2=0x7f8618078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 -- 172.21.15.63:0/1539279586 shutdown_connections 2024-04-25T18:45:03.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.672+0000 7f861e000700 1 -- 172.21.15.63:0/1539279586 wait complete. 2024-04-25T18:45:03.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 Processor -- start 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 -- start start 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806b070 0x7f861819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f861819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8618114e50 con 0x7f861806a750 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8618114cd0 con 0x7f861810f4f0 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.673+0000 7f861e000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8618114fd0 con 0x7f861806b070 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43514/0 (socket says 172.21.15.63:43514) 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 -- 172.21.15.63:0/3757539650 learned_addr learned my addr 172.21.15.63:0/3757539650 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f860ea00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806b070 0x7f861819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:03.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617e00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f861819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 -- 172.21.15.63:0/3757539650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806b070 msgr2=0x7f861819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806b070 0x7f861819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 -- 172.21.15.63:0/3757539650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 msgr2=0x7f861819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f861819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 -- 172.21.15.63:0/3757539650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86181a0370 con 0x7f861806a750 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617e00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f861819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.674+0000 7f8617400700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8608009f60 tx=0x7f860800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.675+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608021070 con 0x7f861806a750 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.675+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8608007ac0 con 0x7f861806a750 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.675+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608005040 con 0x7f861806a750 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.675+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86181a0600 con 0x7f861806a750 2024-04-25T18:45:03.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.675+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86181a7ee0 con 0x7f861806a750 2024-04-25T18:45:03.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.676+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8618101f60 con 0x7f861806a750 2024-04-25T18:45:03.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.678+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8608024070 con 0x7f861806a750 2024-04-25T18:45:03.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.680+0000 7f8614c00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 0x7f85fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:03.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.680+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f860800f020 con 0x7f861806a750 2024-04-25T18:45:03.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.680+0000 7f860ea00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 0x7f85fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:03.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.681+0000 7f860ea00700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 0x7f85fc0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8618101460 tx=0x7f8600009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:03.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.684+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f860802b020 con 0x7f861806a750 2024-04-25T18:45:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.968+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f861802df80 con 0x7f861806a750 2024-04-25T18:45:03.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.969+0000 7f8614c00700 1 -- 172.21.15.63:0/3757539650 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f860809e2a0 con 0x7f861806a750 2024-04-25T18:45:03.970 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:03.970 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T18:45:03.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 msgr2=0x7f85fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:03.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 0x7f85fc0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8618101460 tx=0x7f8600009310 comp rx=0 tx=0).stop 2024-04-25T18:45:03.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 msgr2=0x7f861819b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8608009f60 tx=0x7f860800cd60 comp rx=0 tx=0).stop 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 shutdown_connections 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f85fc0a3c40 0x7f85fc0a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f861810f4f0 0x7f861819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f861806a750 0x7f861819b360 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 --2- 172.21.15.63:0/3757539650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f861806b070 0x7f861819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 >> 172.21.15.63:0/3757539650 conn(0x7f86180788f0 msgr2=0x7f86181109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 shutdown_connections 2024-04-25T18:45:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:03.972+0000 7f861e000700 1 -- 172.21.15.63:0/3757539650 wait complete. 2024-04-25T18:45:04.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:04 smithi063 ceph-mon[28006]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:04.808 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:04 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3757539650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T18:45:05.020 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T18:45:05.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:45:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:04 smithi114 ceph-mon[32988]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:04 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3757539650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T18:45:05.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:04 smithi070 ceph-mon[32820]: pgmap v421: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:05.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:04 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3757539650' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T18:45:05.486 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:06 smithi114 ceph-mon[32988]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:07.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:06 smithi063 ceph-mon[28006]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:07.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:06 smithi070 ceph-mon[32820]: pgmap v422: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:07.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 -- 172.21.15.63:0/2945421205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 msgr2=0x7f88b8105290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 --2- 172.21.15.63:0/2945421205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b8105290 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f88a80099c0 tx=0x7f88a8009cd0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 -- 172.21.15.63:0/2945421205 shutdown_connections 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 --2- 172.21.15.63:0/2945421205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8109f90 0x7f88b8111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 --2- 172.21.15.63:0/2945421205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b8105290 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 --2- 172.21.15.63:0/2945421205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8105860 0x7f88b81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 -- 172.21.15.63:0/2945421205 >> 172.21.15.63:0/2945421205 conn(0x7f88b8100780 msgr2=0x7f88b8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 -- 172.21.15.63:0/2945421205 shutdown_connections 2024-04-25T18:45:07.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.753+0000 7f88bd200700 1 -- 172.21.15.63:0/2945421205 wait complete. 2024-04-25T18:45:07.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.754+0000 7f88bd200700 1 Processor -- start 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.754+0000 7f88bd200700 1 -- start start 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8105860 0x7f88b819b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8109f90 0x7f88b819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8114ea0 con 0x7f88b8104eb0 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8114d20 con 0x7f88b8109f90 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88bd200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8115020 con 0x7f88b8105860 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88b5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88b5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41908/0 (socket says 172.21.15.63:41908) 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88b5e00700 1 -- 172.21.15.63:0/2823663456 learned_addr learned my addr 172.21.15.63:0/2823663456 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88b5400700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8105860 0x7f88b819b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.755+0000 7f88b6800700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8109f90 0x7f88b819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:07.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 -- 172.21.15.63:0/2823663456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8105860 msgr2=0x7f88b819b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:07.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8105860 0x7f88b819b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 -- 172.21.15.63:0/2823663456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8109f90 msgr2=0x7f88b819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:07.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8109f90 0x7f88b819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:07.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 -- 172.21.15.63:0/2823663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b81a03d0 con 0x7f88b8104eb0 2024-04-25T18:45:07.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88b5e00700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88a8005910 tx=0x7f88a8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8026070 con 0x7f88b8104eb0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a8003c70 con 0x7f88b8104eb0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b81a0660 con 0x7f88b8104eb0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.756+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800fa40 con 0x7f88b8104eb0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.757+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b81a7f40 con 0x7f88b8104eb0 2024-04-25T18:45:07.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.758+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b8069730 con 0x7f88b8104eb0 2024-04-25T18:45:07.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.760+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f88a8029070 con 0x7f88b8104eb0 2024-04-25T18:45:07.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.761+0000 7f88aea00700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:07.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.761+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f88a80d0bb0 con 0x7f88b8104eb0 2024-04-25T18:45:07.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.761+0000 7f88b5400700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:07.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.762+0000 7f88b5400700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 0x7f889c0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009680 tx=0x7f88a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:07.766+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88a805dff0 con 0x7f88b8104eb0 2024-04-25T18:45:08.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.051+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88b819c720 con 0x7f88b8104eb0 2024-04-25T18:45:08.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.053+0000 7f88aea00700 1 -- 172.21.15.63:0/2823663456 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7f88a8013070 con 0x7f88b8104eb0 2024-04-25T18:45:08.055 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:08.055 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:45:08.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 msgr2=0x7f889c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:08.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 0x7f889c0a6180 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88a4009680 tx=0x7f88a4009260 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 msgr2=0x7f88b819b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f88a8005910 tx=0x7f88a8004270 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 shutdown_connections 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f889c0a3cc0 0x7f889c0a6180 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f88b8109f90 0x7f88b819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88b8104eb0 0x7f88b819b410 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.056+0000 7f88bd200700 1 --2- 172.21.15.63:0/2823663456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f88b8105860 0x7f88b819b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.057+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 >> 172.21.15.63:0/2823663456 conn(0x7f88b8100780 msgr2=0x7f88b8102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.057+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 shutdown_connections 2024-04-25T18:45:08.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:08.057+0000 7f88bd200700 1 -- 172.21.15.63:0/2823663456 wait complete. 2024-04-25T18:45:08.059 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:45:09.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:45:09.109 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:08 smithi063 ceph-mon[28006]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:09.109 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:08 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2823663456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:08 smithi070 ceph-mon[32820]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:09.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:08 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2823663456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:08 smithi114 ceph-mon[32988]: pgmap v423: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:09.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:08 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2823663456' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:09.499 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:11.119 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:10 smithi114 ceph-mon[32988]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:11.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:10 smithi063 ceph-mon[28006]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:11.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:10 smithi070 ceph-mon[32820]: pgmap v424: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:11.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 -- 172.21.15.63:0/473259052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38109fc0 msgr2=0x7ffa38111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:11.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 --2- 172.21.15.63:0/473259052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38109fc0 0x7ffa38111b90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400b0d0 tx=0x7ffa3400b3e0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 -- 172.21.15.63:0/473259052 shutdown_connections 2024-04-25T18:45:11.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 --2- 172.21.15.63:0/473259052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38109fc0 0x7ffa38111b90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 --2- 172.21.15.63:0/473259052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38104ee0 0x7ffa381052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 --2- 172.21.15.63:0/473259052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 0x7ffa381098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 -- 172.21.15.63:0/473259052 >> 172.21.15.63:0/473259052 conn(0x7ffa38100770 msgr2=0x7ffa38102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:11.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 -- 172.21.15.63:0/473259052 shutdown_connections 2024-04-25T18:45:11.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.722+0000 7ffa40800700 1 -- 172.21.15.63:0/473259052 wait complete. 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 Processor -- start 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 -- start start 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 0x7ffa38077490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 0x7ffa380779d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa38114e80 con 0x7ffa38104ee0 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa38114d00 con 0x7ffa38109fc0 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa40800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa38115000 con 0x7ffa38105890 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa3de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41918/0 (socket says 172.21.15.63:41918) 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa3de00700 1 -- 172.21.15.63:0/1958768719 learned_addr learned my addr 172.21.15.63:0/1958768719 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.724+0000 7ffa3d400700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 0x7ffa38077490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:11.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3e800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 0x7ffa380779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 -- 172.21.15.63:0/1958768719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 msgr2=0x7ffa38077490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 0x7ffa38077490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 -- 172.21.15.63:0/1958768719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 msgr2=0x7ffa380779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 0x7ffa380779d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 -- 172.21.15.63:0/1958768719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa381aba00 con 0x7ffa38104ee0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3e800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 0x7ffa380779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.725+0000 7ffa3de00700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00e9a0 tx=0x7ffa2c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.726+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c00cbd0 con 0x7ffa38104ee0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.726+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2c00cd70 con 0x7ffa38104ee0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.726+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c018660 con 0x7ffa38104ee0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.726+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa381abcf0 con 0x7ffa38104ee0 2024-04-25T18:45:11.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.726+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa381ac220 con 0x7ffa38104ee0 2024-04-25T18:45:11.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.727+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa381063c0 con 0x7ffa38104ee0 2024-04-25T18:45:11.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.729+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ffa2c010430 con 0x7ffa38104ee0 2024-04-25T18:45:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.731+0000 7ffa32a00700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 0x7ffa200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.731+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7ffa2c0d0610 con 0x7ffa38104ee0 2024-04-25T18:45:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.731+0000 7ffa3d400700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 0x7ffa200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:11.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.732+0000 7ffa3d400700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 0x7ffa200a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffa38078710 tx=0x7ffa28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:11.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:11.736+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa2c09cf60 con 0x7ffa38104ee0 2024-04-25T18:45:12.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.015+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa3802df80 con 0x7ffa38104ee0 2024-04-25T18:45:12.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.016+0000 7ffa32a00700 1 -- 172.21.15.63:0/1958768719 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7ffa2c09c860 con 0x7ffa38104ee0 2024-04-25T18:45:12.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:12.018 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 msgr2=0x7ffa200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 0x7ffa200a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffa38078710 tx=0x7ffa28009260 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 msgr2=0x7ffa38076f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c00e9a0 tx=0x7ffa2c00ecb0 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 shutdown_connections 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7ffa200a3cc0 0x7ffa200a6180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa38109fc0 0x7ffa380779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa38104ee0 0x7ffa38076f50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 --2- 172.21.15.63:0/1958768719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa38105890 0x7ffa38077490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.019+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 >> 172.21.15.63:0/1958768719 conn(0x7ffa38100770 msgr2=0x7ffa381012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.020+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 shutdown_connections 2024-04-25T18:45:12.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:12.020+0000 7ffa40800700 1 -- 172.21.15.63:0/1958768719 wait complete. 2024-04-25T18:45:12.022 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:45:12.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:45:13.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:12 smithi063 ceph-mon[28006]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:13.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:12 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1958768719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:13.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:12 smithi070 ceph-mon[32820]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:13.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:12 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1958768719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:13.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:12 smithi114 ceph-mon[32988]: pgmap v425: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:13.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:12 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1958768719' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:13.378 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:15.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:14 smithi063 ceph-mon[28006]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:15.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:14 smithi070 ceph-mon[32820]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:15.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:14 smithi114 ceph-mon[32988]: pgmap v426: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:15.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 -- 172.21.15.63:0/3439977196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48102230 msgr2=0x7fbc4810e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/3439977196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48102230 0x7fbc4810e740 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbc4000b0a0 tx=0x7fbc4000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 -- 172.21.15.63:0/3439977196 shutdown_connections 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/3439977196 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48107870 0x7fbc48107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/3439977196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48102230 0x7fbc4810e740 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/3439977196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 0x7fbc48101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 -- 172.21.15.63:0/3439977196 >> 172.21.15.63:0/3439977196 conn(0x7fbc480fd360 msgr2=0x7fbc480ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 -- 172.21.15.63:0/3439977196 shutdown_connections 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.695+0000 7fbc4ca00700 1 -- 172.21.15.63:0/3439977196 wait complete. 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.696+0000 7fbc4ca00700 1 Processor -- start 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 -- start start 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 0x7fbc48198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48107870 0x7fbc4819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48111a70 con 0x7fbc48107870 2024-04-25T18:45:15.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc481118f0 con 0x7fbc48101870 2024-04-25T18:45:15.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.697+0000 7fbc4ca00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48111bf0 con 0x7fbc48102230 2024-04-25T18:45:15.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc47e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48107870 0x7fbc4819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc47400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 0x7fbc48198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48364/0 (socket says 172.21.15.63:48364) 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 -- 172.21.15.63:0/987591445 learned_addr learned my addr 172.21.15.63:0/987591445 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 -- 172.21.15.63:0/987591445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 msgr2=0x7fbc48198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 0x7fbc48198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 -- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48107870 msgr2=0x7fbc4819d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48107870 0x7fbc4819d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 -- 172.21.15.63:0/987591445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4819dfd0 con 0x7fbc48102230 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.698+0000 7fbc46a00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400e9a0 tx=0x7fbc3400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.699+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3400ca10 con 0x7fbc48102230 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.699+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc3400cbb0 con 0x7fbc48102230 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.699+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34018680 con 0x7fbc48102230 2024-04-25T18:45:15.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.699+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4819e2c0 con 0x7fbc48102230 2024-04-25T18:45:15.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.699+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc481a5b40 con 0x7fbc48102230 2024-04-25T18:45:15.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.700+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc48069730 con 0x7fbc48102230 2024-04-25T18:45:15.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.702+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc34010430 con 0x7fbc48102230 2024-04-25T18:45:15.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.705+0000 7fbc3fe00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 0x7fbc2c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:15.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.705+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fbc340d0860 con 0x7fbc48102230 2024-04-25T18:45:15.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.705+0000 7fbc47400700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 0x7fbc2c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:15.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.706+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc3409d1b0 con 0x7fbc48102230 2024-04-25T18:45:15.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.706+0000 7fbc47400700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 0x7fbc2c0a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009be0 tx=0x7fbc38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:15.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.938+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc4819a320 con 0x7fbc48102230 2024-04-25T18:45:15.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.941+0000 7fbc3fe00700 1 -- 172.21.15.63:0/987591445 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+22736 (secure 0 0 0) 0x7fbc3409cab0 con 0x7fbc48102230 2024-04-25T18:45:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:15.943 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":96,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:43:55.344181+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:45:15.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 msgr2=0x7fbc2c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:15.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 0x7fbc2c0a6180 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009be0 tx=0x7fbc38009260 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 msgr2=0x7fbc48199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400e9a0 tx=0x7fbc3400ecb0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 shutdown_connections 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbc2c0a3cc0 0x7fbc2c0a6180 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbc48107870 0x7fbc4819d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc48101870 0x7fbc48198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 --2- 172.21.15.63:0/987591445 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbc48102230 0x7fbc48199500 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 >> 172.21.15.63:0/987591445 conn(0x7fbc480fd360 msgr2=0x7fbc4806a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 shutdown_connections 2024-04-25T18:45:15.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:15.944+0000 7fbc4ca00700 1 -- 172.21.15.63:0/987591445 wait complete. 2024-04-25T18:45:16.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:45:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:16 smithi063 ceph-mon[28006]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:16 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/987591445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:45:17.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:16 smithi070 ceph-mon[32820]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:17.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:16 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/987591445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:45:17.298 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:17.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:16 smithi114 ceph-mon[32988]: pgmap v427: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:17.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:16 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/987591445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:45:19.039 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:18 smithi063 ceph-mon[28006]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:19.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:18 smithi070 ceph-mon[32820]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:19.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:18 smithi114 ceph-mon[32988]: pgmap v428: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:19.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 -- 172.21.15.63:0/3498042930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8113920 msgr2=0x7fc5a8115d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:19.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 --2- 172.21.15.63:0/3498042930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8113920 0x7fc5a8115d10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000b0a0 tx=0x7fc5a000b3b0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 -- 172.21.15.63:0/3498042930 shutdown_connections 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 --2- 172.21.15.63:0/3498042930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8113920 0x7fc5a8115d10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 --2- 172.21.15.63:0/3498042930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8076f70 0x7fc5a8077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 --2- 172.21.15.63:0/3498042930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a81133e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 -- 172.21.15.63:0/3498042930 >> 172.21.15.63:0/3498042930 conn(0x7fc5a8100770 msgr2=0x7fc5a8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 -- 172.21.15.63:0/3498042930 shutdown_connections 2024-04-25T18:45:19.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.556+0000 7fc5ada00700 1 -- 172.21.15.63:0/3498042930 wait complete. 2024-04-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.557+0000 7fc5ada00700 1 Processor -- start 2024-04-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.557+0000 7fc5ada00700 1 -- start start 2024-04-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:19.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8113920 0x7fc5a81a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a81192a0 con 0x7fc5a8076f70 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a8119120 con 0x7fc5a8113920 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5ada00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a8119420 con 0x7fc5a8077890 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5a6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5a6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47472/0 (socket says 172.21.15.63:47472) 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5a6a00700 1 -- 172.21.15.63:0/1805235388 learned_addr learned my addr 172.21.15.63:0/1805235388 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5a6000700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:19.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.558+0000 7fc5a7400700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8113920 0x7fc5a81a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 -- 172.21.15.63:0/1805235388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 msgr2=0x7fc5a819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 -- 172.21.15.63:0/1805235388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8113920 msgr2=0x7fc5a81a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8113920 0x7fc5a81a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 -- 172.21.15.63:0/1805235388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a81a4720 con 0x7fc5a8076f70 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6000700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5a6a00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc59800eb30 tx=0x7fc59800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59800cbd0 con 0x7fc5a8076f70 2024-04-25T18:45:19.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5980044d0 con 0x7fc5a8076f70 2024-04-25T18:45:19.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598003e30 con 0x7fc5a8076f70 2024-04-25T18:45:19.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.559+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a81a4a10 con 0x7fc5a8076f70 2024-04-25T18:45:19.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.560+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a81ac290 con 0x7fc5a8076f70 2024-04-25T18:45:19.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.561+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a8069730 con 0x7fc5a8076f70 2024-04-25T18:45:19.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.565+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc598003fd0 con 0x7fc5a8076f70 2024-04-25T18:45:19.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.566+0000 7fc59f400700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 0x7fc58c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:19.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.566+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fc5980d0bf0 con 0x7fc5a8076f70 2024-04-25T18:45:19.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.566+0000 7fc5a6000700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 0x7fc58c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:19.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.567+0000 7fc5a6000700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 0x7fc58c0a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8104350 tx=0x7fc59400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:19.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.569+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc59801d060 con 0x7fc5a8076f70 2024-04-25T18:45:19.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.848+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5a81a0a70 con 0x7fc5a8076f70 2024-04-25T18:45:19.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.850+0000 7fc59f400700 1 -- 172.21.15.63:0/1805235388 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7fc598020390 con 0x7fc5a8076f70 2024-04-25T18:45:19.852 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:19.852 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 msgr2=0x7fc58c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 0x7fc58c0a6180 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8104350 tx=0x7fc59400b410 comp rx=0 tx=0).stop 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 msgr2=0x7fc5a819f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc59800eb30 tx=0x7fc59800ee40 comp rx=0 tx=0).stop 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 shutdown_connections 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc58c0a3cc0 0x7fc58c0a6180 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5a8113920 0x7fc5a81a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a8076f70 0x7fc5a819f710 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 --2- 172.21.15.63:0/1805235388 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5a8077890 0x7fc5a819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:19.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.853+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 >> 172.21.15.63:0/1805235388 conn(0x7fc5a8100770 msgr2=0x7fc5a8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:19.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.854+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 shutdown_connections 2024-04-25T18:45:19.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:19.854+0000 7fc5ada00700 1 -- 172.21.15.63:0/1805235388 wait complete. 2024-04-25T18:45:19.857 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:45:20.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T18:42:31.925526+0000', 'modified': '2024-04-25T18:42:57.440972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24533, 'mds_1': 24520, 'mds_2': 24508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/3182056159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 3182056159}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 3182056159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6837/4043589496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 4043589496}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 4043589496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/3172988525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3172988525}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 3172988525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.114:6837/1526734616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 1526734616}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1526734616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.114:6839/2417088150', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 2417088150}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 2417088150}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.70:6839/1944035005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 1944035005}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 1944035005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24533, 24508, 24520]} 2024-04-25T18:45:20.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T18:45:20.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-25T18:45:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:20 smithi063 ceph-mon[28006]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:21.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:20 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1805235388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:21.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:20 smithi070 ceph-mon[32820]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:21.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:20 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1805235388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:21.243 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:21.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:20 smithi114 ceph-mon[32988]: pgmap v429: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:21.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:20 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1805235388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:45:23.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:22 smithi063 ceph-mon[28006]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:23.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:22 smithi070 ceph-mon[32820]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:23.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:22 smithi114 ceph-mon[32988]: pgmap v430: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.354+0000 7f1ded000700 1 -- 172.21.15.63:0/3284068810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de806a750 msgr2=0x7f1de806ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.354+0000 7f1ded000700 1 --2- 172.21.15.63:0/3284068810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de806a750 0x7f1de806ab30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009990 tx=0x7f1dd8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 -- 172.21.15.63:0/3284068810 shutdown_connections 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 --2- 172.21.15.63:0/3284068810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de810f4f0 0x7f1de81118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 --2- 172.21.15.63:0/3284068810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de806a750 0x7f1de806ab30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 --2- 172.21.15.63:0/3284068810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806b070 0x7f1de810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 -- 172.21.15.63:0/3284068810 >> 172.21.15.63:0/3284068810 conn(0x7f1de80788f0 msgr2=0x7f1de8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 -- 172.21.15.63:0/3284068810 shutdown_connections 2024-04-25T18:45:23.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.355+0000 7f1ded000700 1 -- 172.21.15.63:0/3284068810 wait complete. 2024-04-25T18:45:23.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.356+0000 7f1ded000700 1 Processor -- start 2024-04-25T18:45:23.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.356+0000 7f1ded000700 1 -- start start 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806a750 0x7f1de819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 0x7f1de819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 0x7f1de819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1de8114e50 con 0x7f1de810f4f0 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1de8114cd0 con 0x7f1de806b070 2024-04-25T18:45:23.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1ded000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1de8114fd0 con 0x7f1de806a750 2024-04-25T18:45:23.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1de6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 0x7f1de819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1de6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 0x7f1de819fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47482/0 (socket says 172.21.15.63:47482) 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1de6800700 1 -- 172.21.15.63:0/2971390426 learned_addr learned my addr 172.21.15.63:0/2971390426 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1de5400700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 0x7f1de819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.357+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806a750 0x7f1de819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 -- 172.21.15.63:0/2971390426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806a750 msgr2=0x7f1de819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806a750 0x7f1de819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 -- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 msgr2=0x7f1de819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 0x7f1de819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 -- 172.21.15.63:0/2971390426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de81a0370 con 0x7f1de806b070 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1de5400700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 0x7f1de819b8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400e9f0 tx=0x7f1dd400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1ddea00700 1 -- 172.21.15.63:0/2971390426 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd400c9c0 con 0x7f1de806b070 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1ddea00700 1 -- 172.21.15.63:0/2971390426 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd400cb60 con 0x7f1de806b070 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1ddea00700 1 -- 172.21.15.63:0/2971390426 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4018680 con 0x7f1de806b070 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.358+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de81a0660 con 0x7f1de806b070 2024-04-25T18:45:23.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.359+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de81a7ee0 con 0x7f1de806b070 2024-04-25T18:45:23.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.362+0000 7f1ddea00700 1 -- 172.21.15.63:0/2971390426 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1dd4010430 con 0x7f1de806b070 2024-04-25T18:45:23.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.363+0000 7f1df791bb80 1 Processor -- start 2024-04-25T18:45:23.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.363+0000 7f1df791bb80 1 -- start start 2024-04-25T18:45:23.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1ddea00700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1ddea00700 1 -- 172.21.15.63:0/2971390426 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f1dd40d05e0 con 0x7f1de806b070 2024-04-25T18:45:23.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1de6800700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287afc10 0x5643287ba570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287bcea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564328702e40 con 0x5643287afc10 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643286a96e0 con 0x5643287baab0 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1df791bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643286fee00 con 0x5643287af2f0 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1de5e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1de5e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45106/0 (socket says 172.21.15.63:45106) 2024-04-25T18:45:23.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.364+0000 7f1de5e00700 1 -- 172.21.15.63:0/3143515397 learned_addr learned my addr 172.21.15.63:0/3143515397 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de6800700 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287bcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5400700 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287afc10 0x5643287ba570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de6800700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009fd0 tx=0x7f1de0009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5e00700 1 -- 172.21.15.63:0/3143515397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 msgr2=0x5643287bcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5e00700 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287bcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5e00700 1 -- 172.21.15.63:0/3143515397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287afc10 msgr2=0x5643287ba570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5e00700 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287afc10 0x5643287ba570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.365+0000 7f1de5e00700 1 -- 172.21.15.63:0/3143515397 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56432869f630 con 0x5643287af2f0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.366+0000 7f1de5e00700 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80096a0 tx=0x7f1dd80096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8e76fd0e12216dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.366+0000 7f1dd3400700 1 -- 172.21.15.63:0/3143515397 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8027070 con 0x5643287af2f0 2024-04-25T18:45:23.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.366+0000 7f1dd3400700 1 -- 172.21.15.63:0/3143515397 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd80056d0 con 0x5643287af2f0 2024-04-25T18:45:23.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.366+0000 7f1dd3400700 1 -- 172.21.15.63:0/3143515397 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8003b40 con 0x5643287af2f0 2024-04-25T18:45:23.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 -- 172.21.15.63:0/3143515397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 msgr2=0x5643287af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80096a0 tx=0x7f1dd80096d0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 -- 172.21.15.63:0/3143515397 shutdown_connections 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287af2f0 0x5643287af6d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 --2- 172.21.15.63:0/3143515397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287afc10 0x5643287ba570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 -- 172.21.15.63:0/3143515397 >> 172.21.15.63:0/3143515397 conn(0x5643286de550 msgr2=0x5643287ab790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 -- 172.21.15.63:0/3143515397 shutdown_connections 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.367+0000 7f1df791bb80 1 -- 172.21.15.63:0/3143515397 wait complete. 2024-04-25T18:45:23.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 -- start start 2024-04-25T18:45:23.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 Processor -- start 2024-04-25T18:45:23.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287ba1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643287bd540 con 0x5643287baab0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643286bc810 con 0x5643287af2f0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.368+0000 7f1df791bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643287bd6c0 con 0x5643287afc10 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1df791bb80 10 client.0 _ll_register_callbacks cb 0x56432869b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47492/0 (socket says 172.21.15.63:47492) 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de6800700 1 -- 172.21.15.63:0/2380306997 learned_addr learned my addr 172.21.15.63:0/2380306997 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287ba1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 -- 172.21.15.63:0/2380306997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 msgr2=0x5643287ba1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287ba1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 -- 172.21.15.63:0/2380306997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 msgr2=0x5643287b7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287b7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.369+0000 7f1de5400700 1 -- 172.21.15.63:0/2380306997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564328650880 con 0x5643287afc10 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1de5400700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287b74e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1dd409c530 tx=0x7f1dd4087390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16970a68d1fd162d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1dc7e00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40a0be0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f1dd40a5010 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dd80056d0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1dd40a54a0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1df791bb80 10 client.24892 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564328699aa0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.370+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd80056d0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1dd40d00b0 con 0x5643287afc10 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 10 client.24892 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 10 client.24892 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 20 client.24892 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x5643287ca280 0x5643287cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 4 client.24892 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:45:23.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564328710630 con 0x5643287ca280 2024-04-25T18:45:23.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.371+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x5643287ca280 0x5643287cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.372+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x5643287ca280 0x5643287cb6f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b68e1ef97f18a55 server_cookie=8f92f89435b1f598 in_seq=0 out_seq=0 2024-04-25T18:45:23.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.372+0000 7f1dc7e00700 10 client.24892 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2380306997 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564328710630 con 0x5643287ca280 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1dc7e00700 10 client.24892 handle_command_reply: tid=0 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 client.24892 shutdown 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 msgr2=0x5643287b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287b74e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1dd409c530 tx=0x7f1dd4087390 comp rx=0 tx=0).stop 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 shutdown_connections 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x5643287ca280 0x5643287cb6f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287b7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287ba1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2380306997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287b74e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1dc7e00700 0 client.24892 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 >> 172.21.15.63:0/2380306997 conn(0x5643286de550 msgr2=0x5643287aa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 shutdown_connections 2024-04-25T18:45:23.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.373+0000 7f1df791bb80 1 -- 172.21.15.63:0/2380306997 wait complete. 2024-04-25T18:45:23.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.374+0000 7f1df791bb80 20 client.24892 trim_cache size 0 max 16384 2024-04-25T18:45:23.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.396+0000 7f1df791bb80 1 Processor -- start 2024-04-25T18:45:23.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.396+0000 7f1df791bb80 1 -- start start 2024-04-25T18:45:23.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.396+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.396+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.396+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287ab610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1df791bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643287c9a30 con 0x5643287baab0 2024-04-25T18:45:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1df791bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643287ca100 con 0x5643287af2f0 2024-04-25T18:45:23.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1df791bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643287ca7d0 con 0x5643287afc10 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47626/0 (socket says 172.21.15.63:47626) 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 -- 172.21.15.63:0/4037411746 learned_addr learned my addr 172.21.15.63:0/4037411746 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de6800700 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 -- 172.21.15.63:0/4037411746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 msgr2=0x5643287ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 -- 172.21.15.63:0/4037411746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 msgr2=0x5643287ab610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287ab610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5e00700 1 -- 172.21.15.63:0/4037411746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643287b3110 con 0x5643287af2f0 2024-04-25T18:45:23.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.397+0000 7f1de5400700 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:23.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.398+0000 7f1de5e00700 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80036f0 tx=0x7f1dd8003720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3d469e40d98fd25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.398+0000 7f1dd2a00700 1 -- 172.21.15.63:0/4037411746 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8037070 con 0x5643287af2f0 2024-04-25T18:45:23.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.398+0000 7f1dd2a00700 1 -- 172.21.15.63:0/4037411746 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd8028040 con 0x5643287af2f0 2024-04-25T18:45:23.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.398+0000 7f1dd2a00700 1 -- 172.21.15.63:0/4037411746 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8032e60 con 0x5643287af2f0 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- 172.21.15.63:0/4037411746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 msgr2=0x5643287b2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80036f0 tx=0x7f1dd8003720 comp rx=0 tx=0).stop 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- 172.21.15.63:0/4037411746 shutdown_connections 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287baab0 0x5643287ab610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287af2f0 0x5643287b2bd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 --2- 172.21.15.63:0/4037411746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- 172.21.15.63:0/4037411746 >> 172.21.15.63:0/4037411746 conn(0x5643286de550 msgr2=0x5643287b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- 172.21.15.63:0/4037411746 shutdown_connections 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- 172.21.15.63:0/4037411746 wait complete. 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.399+0000 7f1df791bb80 1 -- start start 2024-04-25T18:45:23.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 Processor -- start 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287af2f0 0x5643287d4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643287cb570 con 0x5643287af2f0 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643287caea0 con 0x5643287baab0 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643287cbc40 con 0x5643287afc10 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1df791bb80 10 client.0 _ll_register_callbacks cb 0x56432869b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:23.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1de5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1de5400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45136/0 (socket says 172.21.15.63:45136) 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1de6800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1de5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287af2f0 0x5643287d4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.400+0000 7f1de5400700 1 -- 172.21.15.63:0/2620690466 learned_addr learned my addr 172.21.15.63:0/2620690466 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 -- 172.21.15.63:0/2620690466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 msgr2=0x5643287cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 -- 172.21.15.63:0/2620690466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287af2f0 msgr2=0x5643287d4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287af2f0 0x5643287d4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 -- 172.21.15.63:0/2620690466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643287ce370 con 0x5643287afc10 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1de5400700 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1dd409c7c0 tx=0x7f1dd4098ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6edfe3049de52a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1dc7e00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd40a0be0 con 0x5643287afc10 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f1dd40672e0 con 0x5643287afc10 2024-04-25T18:45:23.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.401+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dd8028040 con 0x5643287afc10 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1dd409db60 con 0x5643287afc10 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 10 client.24898 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564328831f10 con 0x5643287afc10 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1dd8028040 con 0x5643287afc10 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f1dd4098580 con 0x5643287afc10 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 10 client.24898 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 10 client.24898 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 20 client.24898 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x564328836b50 0x564328835860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 4 client.24898 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:45:23.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.402+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "status"}) -- 0x56432870a6d0 con 0x564328836b50 2024-04-25T18:45:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x564328836b50 0x564328835860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1de5e00700 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x564328836b50 0x564328835860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=603a1a250c2e28a5 server_cookie=636342db05e9127f in_seq=0 out_seq=0 2024-04-25T18:45:23.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1dc7e00700 10 client.24898 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1dc7e00700 1 -- 172.21.15.63:0/2620690466 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56432870a6d0 con 0x564328836b50 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1dc7e00700 10 client.24898 handle_command_reply: tid=0 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.403+0000 7f1df791bb80 1 client.24898 shutdown 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 msgr2=0x5643287d50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1dd409c7c0 tx=0x7f1dd4098ad0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 shutdown_connections 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x564328836b50 0x564328835860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643287baab0 0x5643287cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643287af2f0 0x5643287d4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1dc7e00700 0 client.24898 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 --2- 172.21.15.63:0/2620690466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643287afc10 0x5643287d50b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 >> 172.21.15.63:0/2620690466 conn(0x5643286de550 msgr2=0x5643287b95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 shutdown_connections 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 1 -- 172.21.15.63:0/2620690466 wait complete. 2024-04-25T18:45:23.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.404+0000 7f1df791bb80 20 client.24898 trim_cache size 0 max 16384 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 2, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24508, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 152.76153579199999, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 212.55207675299999 2024-04-25T18:45:23.408 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 msgr2=0x7f1dcc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009fd0 tx=0x7f1de0009920 comp rx=0 tx=0).stop 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 msgr2=0x7f1de819b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 0x7f1de819b8a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400e9f0 tx=0x7f1dd400ed00 comp rx=0 tx=0).stop 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 shutdown_connections 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1dcc0a3c40 0x7f1dcc0a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1de806a750 0x7f1de819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1de810f4f0 0x7f1de819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 --2- 172.21.15.63:0/2971390426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1de806b070 0x7f1de819b8a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 >> 172.21.15.63:0/2971390426 conn(0x7f1de80788f0 msgr2=0x7f1de81109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 shutdown_connections 2024-04-25T18:45:23.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:23.405+0000 7f1ded000700 1 -- 172.21.15.63:0/2971390426 wait complete. 2024-04-25T18:45:24.303 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T18:45:24.770 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:25.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:24 smithi063 ceph-mon[28006]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:25.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:24 smithi070 ceph-mon[32820]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:25.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:24 smithi114 ceph-mon[32988]: pgmap v431: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 -- 172.21.15.63:0/2338163466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f089806ac90 msgr2=0x7f089806b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 --2- 172.21.15.63:0/2338163466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f089806ac90 0x7f089806b0f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0884009990 tx=0x7f0884009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 -- 172.21.15.63:0/2338163466 shutdown_connections 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 --2- 172.21.15.63:0/2338163466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898105480 0x7f0898105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 --2- 172.21.15.63:0/2338163466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0898107970 0x7f089806a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 --2- 172.21.15.63:0/2338163466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f089806ac90 0x7f089806b0f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 -- 172.21.15.63:0/2338163466 >> 172.21.15.63:0/2338163466 conn(0x7f08980788f0 msgr2=0x7f0898078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 -- 172.21.15.63:0/2338163466 shutdown_connections 2024-04-25T18:45:26.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.836+0000 7f089f000700 1 -- 172.21.15.63:0/2338163466 wait complete. 2024-04-25T18:45:26.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.837+0000 7f089f000700 1 Processor -- start 2024-04-25T18:45:26.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 -- start start 2024-04-25T18:45:26.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 0x7f089819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898107970 0x7f08981a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0898119230 con 0x7f089806ac90 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08981190b0 con 0x7f0898107970 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f089f000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f08981193b0 con 0x7f0898105480 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f0897e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f0897400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 0x7f089819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f0897e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47522/0 (socket says 172.21.15.63:47522) 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.838+0000 7f0897e00700 1 -- 172.21.15.63:0/3302730391 learned_addr learned my addr 172.21.15.63:0/3302730391 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f089d000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898107970 0x7f08981a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 -- 172.21.15.63:0/3302730391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898107970 msgr2=0x7f08981a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898107970 0x7f08981a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 -- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 msgr2=0x7f089819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 -- 172.21.15.63:0/3302730391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08981a4730 con 0x7f0898105480 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897e00700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.839+0000 7f0897400700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 0x7f089819fc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08840098e0 tx=0x7f08840057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.840+0000 7f0894c00700 1 -- 172.21.15.63:0/3302730391 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884027070 con 0x7f0898105480 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.840+0000 7f0894c00700 1 -- 172.21.15.63:0/3302730391 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0884022470 con 0x7f0898105480 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.840+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08981a49c0 con 0x7f0898105480 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.840+0000 7f0894c00700 1 -- 172.21.15.63:0/3302730391 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884017650 con 0x7f0898105480 2024-04-25T18:45:26.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.840+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08981ac2a0 con 0x7f0898105480 2024-04-25T18:45:26.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.843+0000 7f0894c00700 1 -- 172.21.15.63:0/3302730391 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f088402a070 con 0x7f0898105480 2024-04-25T18:45:26.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.844+0000 7f08a9a26b80 1 Processor -- start 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0894c00700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 0x7f087c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 -- start start 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0894c00700 1 -- 172.21.15.63:0/3302730391 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f08840d10b0 con 0x7f0898105480 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f089d000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 0x7f087c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 0x560f4482d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f4482def0 0x560f4482e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560f4477fe40 con 0x560f4482def0 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f447266e0 con 0x560f448340a0 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f4477be00 con 0x560f44833780 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0897e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0897e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45176/0 (socket says 172.21.15.63:45176) 2024-04-25T18:45:26.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0897e00700 1 -- 172.21.15.63:0/2015627827 learned_addr learned my addr 172.21.15.63:0/2015627827 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:26.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f089d000700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f4482def0 0x560f4482e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.845+0000 7f0897400700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 0x560f4482d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 -- 172.21.15.63:0/2015627827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 msgr2=0x560f44833b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 -- 172.21.15.63:0/2015627827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f4482def0 msgr2=0x560f4482e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f4482def0 0x560f4482e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 -- 172.21.15.63:0/2015627827 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f4471c630 con 0x560f448340a0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897e00700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f089d000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 0x7f087c0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f089000f870 tx=0x7f089000fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.846+0000 7f0897400700 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 0x560f4482d9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f088409d320 tx=0x7f0884009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=124b3a29488260ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.847+0000 7f088d600700 1 -- 172.21.15.63:0/2015627827 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08840a1740 con 0x560f448340a0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.847+0000 7f088d600700 1 -- 172.21.15.63:0/2015627827 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08840a5f70 con 0x560f448340a0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.847+0000 7f088d600700 1 -- 172.21.15.63:0/2015627827 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088409d5f0 con 0x560f448340a0 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.847+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2015627827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 msgr2=0x560f4482d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.847+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 0x560f4482d9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f088409d320 tx=0x7f0884009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2015627827 shutdown_connections 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44833780 0x560f44833b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f4482def0 0x560f4482e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2015627827 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f448340a0 0x560f4482d9b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2015627827 >> 172.21.15.63:0/2015627827 conn(0x560f4475bf00 msgr2=0x560f448288a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2015627827 shutdown_connections 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2015627827 wait complete. 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.848+0000 7f08a9a26b80 1 -- start start 2024-04-25T18:45:26.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 Processor -- start 2024-04-25T18:45:26.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 0x560f44839cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44836fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560f4483a6d0 con 0x560f44833780 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f44739810 con 0x560f4482def0 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f4483a850 con 0x560f44837510 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f08a9a26b80 10 client.0 _ll_register_callbacks cb 0x560f447186d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44836fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.849+0000 7f0897e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 0x560f44839cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44836fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47532/0 (socket says 172.21.15.63:47532) 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45188/0 (socket says 172.21.15.63:45188) 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 -- 172.21.15.63:0/2339804712 learned_addr learned my addr 172.21.15.63:0/2339804712 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f0897e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 0x560f44839cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47690/0 (socket says 172.21.15.63:47690) 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 -- 172.21.15.63:0/2339804712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 msgr2=0x560f44839cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 0x560f44839cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 -- 172.21.15.63:0/2339804712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 msgr2=0x560f44836fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44836fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 -- 172.21.15.63:0/2339804712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f446cd880 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f089d000700 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08900060a0 tx=0x7f0890004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=859be0a2e3046b10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.850+0000 7f0882000700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890008920 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f0890008ac0 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08840a5f70 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0890032030 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f08a9a26b80 10 client.24910 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560f44716aa0 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08840a5f70 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.851+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f089001a440 con 0x560f44837510 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 10 client.24910 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 10 client.24910 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:45:26.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 20 client.24910 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:26.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f44847740 0x560f44848bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 4 client.24910 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:45:26.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f4478d630 con 0x560f44847740 2024-04-25T18:45:26.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.852+0000 7f0897e00700 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f44847740 0x560f44848bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.853+0000 7f0897e00700 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f44847740 0x560f44848bb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fa54f65dd6be040 server_cookie=43251631e3150a47 in_seq=0 out_seq=0 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.853+0000 7f0882000700 10 client.24910 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f0882000700 1 -- 172.21.15.63:0/2339804712 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f4478d630 con 0x560f44847740 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f0882000700 10 client.24910 handle_command_reply: tid=0 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 client.24910 shutdown 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 msgr2=0x560f44837990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f08900060a0 tx=0x7f0890004b40 comp rx=0 tx=0).stop 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 shutdown_connections 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f44847740 0x560f44848bb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f44837510 0x560f44837990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f4482def0 0x560f44839cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f0882000700 0 client.24910 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/2339804712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44836fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 >> 172.21.15.63:0/2339804712 conn(0x560f4475bf00 msgr2=0x560f44827e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 shutdown_connections 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 1 -- 172.21.15.63:0/2339804712 wait complete. 2024-04-25T18:45:26.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.854+0000 7f08a9a26b80 20 client.24910 trim_cache size 0 max 16384 2024-04-25T18:45:26.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 Processor -- start 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 -- start start 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f4482d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f44852010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560f44846ef0 con 0x560f44833780 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f448475c0 con 0x560f44837510 2024-04-25T18:45:26.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f44847c90 con 0x560f4482def0 2024-04-25T18:45:26.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f0897e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f4482d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.878+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47540/0 (socket says 172.21.15.63:47540) 2024-04-25T18:45:26.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f089d000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f44852010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 -- 172.21.15.63:0/3759352665 learned_addr learned my addr 172.21.15.63:0/3759352665 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 -- 172.21.15.63:0/3759352665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 msgr2=0x560f4482d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f4482d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 -- 172.21.15.63:0/3759352665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 msgr2=0x560f44852010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f44852010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 -- 172.21.15.63:0/3759352665 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f44835540 con 0x560f44833780 2024-04-25T18:45:26.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.879+0000 7f0897400700 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x560f4483a560 tx=0x7f0884005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d37ff09d9b671c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.880+0000 7f088cc00700 1 -- 172.21.15.63:0/3759352665 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08840a5f70 con 0x560f44833780 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.880+0000 7f088cc00700 1 -- 172.21.15.63:0/3759352665 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08840a1740 con 0x560f44833780 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.880+0000 7f088cc00700 1 -- 172.21.15.63:0/3759352665 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088409d5f0 con 0x560f44833780 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.880+0000 7f08a9a26b80 1 -- 172.21.15.63:0/3759352665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 msgr2=0x560f448285d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.880+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x560f4483a560 tx=0x7f0884005040 comp rx=0 tx=0).stop 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 -- 172.21.15.63:0/3759352665 shutdown_connections 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f4482d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f44852010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/3759352665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f448285d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 -- 172.21.15.63:0/3759352665 >> 172.21.15.63:0/3759352665 conn(0x560f4475bf00 msgr2=0x560f448358c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 -- 172.21.15.63:0/3759352665 shutdown_connections 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 -- 172.21.15.63:0/3759352665 wait complete. 2024-04-25T18:45:26.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 -- start start 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.881+0000 7f08a9a26b80 1 Processor -- start 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f44852030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f4484cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560f44848a30 con 0x560f44833780 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560f44848360 con 0x560f44837510 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560f44849100 con 0x560f4482def0 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f08a9a26b80 10 client.0 _ll_register_callbacks cb 0x560f447186d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f0897e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f44852030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f0897400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47550/0 (socket says 172.21.15.63:47550) 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f0897400700 1 -- 172.21.15.63:0/1272162141 learned_addr learned my addr 172.21.15.63:0/1272162141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:26.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.882+0000 7f089d000700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f4484cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 -- 172.21.15.63:0/1272162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 msgr2=0x560f4484cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f4484cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 -- 172.21.15.63:0/1272162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 msgr2=0x560f44852570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 -- 172.21.15.63:0/1272162141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f4484d820 con 0x560f4482def0 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897400700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0897e00700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f44852030 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x560f44849d30 tx=0x7f0888003ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e0a6412603ee9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:26.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0882000700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888004a30 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08840a1740 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f0888004bd0 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.883+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f088802f050 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 10 client.24916 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x560f448af3b0 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08840a1740 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f0888005330 con 0x560f4482def0 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 10 client.24916 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 10 client.24916 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 20 client.24916 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f448b3f80 0x560f448b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 4 client.24916 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:45:26.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.884+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "status"}) -- 0x560f447876d0 con 0x560f448b3f80 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f0897400700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f448b3f80 0x560f448b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f0897400700 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f448b3f80 0x560f448b2c90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=590cb8e5404c44c8 server_cookie=f6e715f89f0e52ba in_seq=0 out_seq=0 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f0882000700 10 client.24916 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f0882000700 1 -- 172.21.15.63:0/1272162141 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560f447876d0 con 0x560f448b3f80 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f0882000700 10 client.24916 handle_command_reply: tid=0 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.885+0000 7f08a9a26b80 1 client.24916 shutdown 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 msgr2=0x560f44852030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f44852030 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x560f44849d30 tx=0x7f0888003ac0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 shutdown_connections 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x560f448b3f80 0x560f448b2c90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560f4482def0 0x560f44852030 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560f44837510 0x560f4484cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 --2- 172.21.15.63:0/1272162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560f44833780 0x560f44852570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f0882000700 0 client.24916 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:45:26.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 >> 172.21.15.63:0/1272162141 conn(0x560f4475bf00 msgr2=0x560f44834490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 shutdown_connections 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 1 -- 172.21.15.63:0/1272162141 wait complete. 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.886+0000 7f08a9a26b80 20 client.24916 trim_cache size 0 max 16384 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 1, 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24520, 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:45:26.891 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 157.25496272500001, 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 210.74016245300001 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 msgr2=0x7f087c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 0x7f087c0a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f089000f870 tx=0x7f089000fc30 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 msgr2=0x7f089819fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 0x7f089819fc60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08840098e0 tx=0x7f08840057c0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 shutdown_connections 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f087c0a3cc0 0x7f087c0a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0898107970 0x7f08981a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f089806ac90 0x7f089819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 --2- 172.21.15.63:0/3302730391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0898105480 0x7f089819fc60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 >> 172.21.15.63:0/3302730391 conn(0x7f08980788f0 msgr2=0x7f0898104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 shutdown_connections 2024-04-25T18:45:26.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:26.887+0000 7f089f000700 1 -- 172.21.15.63:0/3302730391 wait complete. 2024-04-25T18:45:26.966 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:26 smithi063 ceph-mon[28006]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:27.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:26 smithi070 ceph-mon[32820]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:27.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:26 smithi114 ceph-mon[32988]: pgmap v432: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:28.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T18:45:28.462 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:28 smithi063 ceph-mon[28006]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:29.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:28 smithi070 ceph-mon[32820]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:29.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:28 smithi114 ceph-mon[32988]: pgmap v433: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 -- 172.21.15.63:0/1202807476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c074c70 msgr2=0x7f6c1c075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 --2- 172.21.15.63:0/1202807476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c074c70 0x7f6c1c075050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009990 tx=0x7f6c0c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 -- 172.21.15.63:0/1202807476 shutdown_connections 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 --2- 172.21.15.63:0/1202807476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c1118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 --2- 172.21.15.63:0/1202807476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c074c70 0x7f6c1c075050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 --2- 172.21.15.63:0/1202807476 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c075590 0x7f6c1c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 -- 172.21.15.63:0/1202807476 >> 172.21.15.63:0/1202807476 conn(0x7f6c1c079c00 msgr2=0x7f6c1c07a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 -- 172.21.15.63:0/1202807476 shutdown_connections 2024-04-25T18:45:30.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.777+0000 7f6c22200700 1 -- 172.21.15.63:0/1202807476 wait complete. 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.779+0000 7f6c22200700 1 Processor -- start 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.779+0000 7f6c22200700 1 -- start start 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.779+0000 7f6c22200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c22200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c075590 0x7f6c1c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c22200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c22200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c114df0 con 0x7f6c1c075590 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c22200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c114c70 con 0x7f6c1c074c70 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c22200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c114f70 con 0x7f6c1c10f4b0 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:51404/0 (socket says 172.21.15.63:51404) 2024-04-25T18:45:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 -- 172.21.15.63:0/2522775163 learned_addr learned my addr 172.21.15.63:0/2522775163 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c075590 0x7f6c1c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1be00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 -- 172.21.15.63:0/2522775163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 msgr2=0x7f6c1c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 -- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c075590 msgr2=0x7f6c1c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c075590 0x7f6c1c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1b400700 1 -- 172.21.15.63:0/2522775163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c1a03a0 con 0x7f6c1c074c70 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.780+0000 7f6c1be00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c1b400700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0057f0 tx=0x7f6c0c00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c13e00700 1 -- 172.21.15.63:0/2522775163 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c026070 con 0x7f6c1c074c70 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c1a0630 con 0x7f6c1c074c70 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c13e00700 1 -- 172.21.15.63:0/2522775163 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c0c021410 con 0x7f6c1c074c70 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c1a7f10 con 0x7f6c1c074c70 2024-04-25T18:45:30.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.781+0000 7f6c13e00700 1 -- 172.21.15.63:0/2522775163 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c017650 con 0x7f6c1c074c70 2024-04-25T18:45:30.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.784+0000 7f6c13e00700 1 -- 172.21.15.63:0/2522775163 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6c0c029070 con 0x7f6c1c074c70 2024-04-25T18:45:30.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.786+0000 7f6c13e00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 0x7f6c000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.786+0000 7f6c2cb7fb80 1 Processor -- start 2024-04-25T18:45:30.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.786+0000 7f6c13e00700 1 -- 172.21.15.63:0/2522775163 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f6c0c0d1070 con 0x7f6c1c074c70 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.786+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 0x7f6c000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.786+0000 7f6c2cb7fb80 1 -- start start 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f761db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f7618330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5629f7569e40 con 0x5629f761e0a0 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629f75106e0 con 0x5629f7617ef0 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5629f7565e00 con 0x5629f761d780 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45470/0 (socket says 172.21.15.63:45470) 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1931780124 learned_addr learned my addr 172.21.15.63:0/1931780124 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1b400700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f761db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f7618330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.787+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 0x7f6c000a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c1111b0 tx=0x7f6c08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 -- 172.21.15.63:0/1931780124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 msgr2=0x5629f761db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f761db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 -- 172.21.15.63:0/1931780124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 msgr2=0x5629f76179b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 -- 172.21.15.63:0/1931780124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f7506630 con 0x5629f7617ef0 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:30.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f7618330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6c140093a0 tx=0x7f6c14009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3caccbb9f65d5e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c10c00700 1 -- 172.21.15.63:0/1931780124 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14019030 con 0x5629f7617ef0 2024-04-25T18:45:30.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c10c00700 1 -- 172.21.15.63:0/1931780124 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c14004bb0 con 0x5629f7617ef0 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.788+0000 7f6c10c00700 1 -- 172.21.15.63:0/1931780124 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14005020 con 0x5629f7617ef0 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1931780124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 msgr2=0x5629f7618330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f7618330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6c140093a0 tx=0x7f6c14009920 comp rx=0 tx=0).stop 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1931780124 shutdown_connections 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f761db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f7618330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1931780124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f761e0a0 0x5629f76179b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1931780124 >> 172.21.15.63:0/1931780124 conn(0x5629f7545f00 msgr2=0x5629f76128a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1931780124 shutdown_connections 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.789+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1931780124 wait complete. 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.790+0000 7f6c2cb7fb80 1 -- start start 2024-04-25T18:45:30.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.790+0000 7f6c2cb7fb80 1 Processor -- start 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f7620f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f76218d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5629f7624920 con 0x5629f7617ef0 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629f7523810 con 0x5629f7621450 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5629f7624aa0 con 0x5629f761d780 2024-04-25T18:45:30.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c2cb7fb80 10 client.0 _ll_register_callbacks cb 0x5629f75026d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f76218d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45472/0 (socket says 172.21.15.63:45472) 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 -- 172.21.15.63:0/4049473896 learned_addr learned my addr 172.21.15.63:0/4049473896 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f7620f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 -- 172.21.15.63:0/4049473896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 msgr2=0x5629f7620f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f7620f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 -- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 msgr2=0x5629f76218d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f76218d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.791+0000 7f6c1b400700 1 -- 172.21.15.63:0/4049473896 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f74b7880 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f7620f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c1be00700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f76218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c1b400700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0ceeb0 tx=0x7f6c0c09d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6a89e2e55c5aa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c05600700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c00fd10 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f6c0c0050c0 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c14004bb0 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6c0c09d3b0 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c2cb7fb80 10 client.14784 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.792+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5629f7500aa0 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.793+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c14004bb0 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.793+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6c0c0bc130 con 0x5629f7617ef0 2024-04-25T18:45:30.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 10 client.14784 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 10 client.14784 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 20 client.14784 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f76319e0 0x5629f7632e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 4 client.14784 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629f7577630 con 0x5629f76319e0 2024-04-25T18:45:30.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f76319e0 0x5629f7632e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f76319e0 0x5629f7632e50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64402e6ed251a684 server_cookie=a0f267ea9142809 in_seq=0 out_seq=0 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.794+0000 7f6c05600700 10 client.14784 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.795+0000 7f6c05600700 1 -- 172.21.15.63:0/4049473896 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5629f7577630 con 0x5629f76319e0 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.795+0000 7f6c05600700 10 client.14784 handle_command_reply: tid=0 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.795+0000 7f6c2cb7fb80 1 client.14784 shutdown 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 msgr2=0x5629f7623c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0ceeb0 tx=0x7f6c0c09d8f0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 shutdown_connections 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f76319e0 0x5629f7632e50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f76218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f7623c10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/4049473896 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f7620f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c05600700 0 client.14784 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 >> 172.21.15.63:0/4049473896 conn(0x5629f7545f00 msgr2=0x5629f7611e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 shutdown_connections 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/4049473896 wait complete. 2024-04-25T18:45:30.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.796+0000 7f6c2cb7fb80 20 client.14784 trim_cache size 0 max 16384 2024-04-25T18:45:30.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 Processor -- start 2024-04-25T18:45:30.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 -- start start 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f761d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 0x5629f762dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5629f7631190 con 0x5629f7621450 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629f7631860 con 0x5629f7617ef0 2024-04-25T18:45:30.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5629f7631f30 con 0x5629f761d780 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:35758/0 (socket says 172.21.15.63:35758) 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c1be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 0x5629f762dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c1b400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f761d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.819+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1187784569 learned_addr learned my addr 172.21.15.63:0/1187784569 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:30.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1187784569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 msgr2=0x5629f761d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f761d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1187784569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 msgr2=0x5629f762dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 0x5629f762dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1187784569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f7618ae0 con 0x5629f761d780 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 0x5629f762dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:30.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6c080117d0 tx=0x7f6c08014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c249eac67c84c8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c07e00700 1 -- 172.21.15.63:0/1187784569 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08004030 con 0x5629f761d780 2024-04-25T18:45:30.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c07e00700 1 -- 172.21.15.63:0/1187784569 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c08004830 con 0x5629f761d780 2024-04-25T18:45:30.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.820+0000 7f6c07e00700 1 -- 172.21.15.63:0/1187784569 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c080053e0 con 0x5629f761d780 2024-04-25T18:45:30.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1187784569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 msgr2=0x5629f76218a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6c080117d0 tx=0x7f6c08014000 comp rx=0 tx=0).stop 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1187784569 shutdown_connections 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7621450 0x5629f762dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7617ef0 0x5629f761d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1187784569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f76218a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1187784569 >> 172.21.15.63:0/1187784569 conn(0x5629f7545f00 msgr2=0x5629f761e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1187784569 shutdown_connections 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.821+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1187784569 wait complete. 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.822+0000 7f6c2cb7fb80 1 -- start start 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.822+0000 7f6c2cb7fb80 1 Processor -- start 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.822+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f763c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.822+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c2cb7fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f7637280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5629f7632cd0 con 0x5629f7617ef0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5629f7632600 con 0x5629f7621450 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c2cb7fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5629f76333a0 con 0x5629f761d780 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c2cb7fb80 10 client.0 _ll_register_callbacks cb 0x5629f75026d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:35766/0 (socket says 172.21.15.63:35766) 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1483305000 learned_addr learned my addr 172.21.15.63:0/1483305000 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1b400700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f763c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1be00700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f7637280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 msgr2=0x5629f7637280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f7637280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1483305000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 msgr2=0x5629f763c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f763c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.823+0000 7f6c1aa00700 1 -- 172.21.15.63:0/1483305000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f7637a70 con 0x5629f761d780 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c1aa00700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6c0802e040 tx=0x7f6c08004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4786ee91391678e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c05600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:45:30.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08004c50 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f6c08004040 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c08004830 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f6c08004240 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c2cb7fb80 10 client.24934 fetch_fsmap learned FSMap version 23 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5629f7699570 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.824+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c08004830 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f6c0803f070 con 0x5629f761d780 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 10 client.24934 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 10 client.24934 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 20 client.24934 populate_metadata read hostname 'smithi063' 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f769dd80 0x5629f769ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 4 client.24934 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:45:30.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "status"}) -- 0x5629f75716d0 con 0x5629f769dd80 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.825+0000 7f6c1b400700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f769dd80 0x5629f769ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c1b400700 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f769dd80 0x5629f769ca90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1928cb6ba5e36cae server_cookie=2117c1c3b905b241 in_seq=0 out_seq=0 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c05600700 10 client.24934 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c05600700 1 -- 172.21.15.63:0/1483305000 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5629f75716d0 con 0x5629f769dd80 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c05600700 10 client.24934 handle_command_reply: tid=0 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 client.24934 shutdown 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 msgr2=0x5629f763c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6c0802e040 tx=0x7f6c08004a60 comp rx=0 tx=0).stop 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 shutdown_connections 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5629f769dd80 0x5629f769ca90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5629f7621450 0x5629f7637280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5629f7617ef0 0x5629f763c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 --2- 172.21.15.63:0/1483305000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5629f761d780 0x5629f763c810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c05600700 0 client.24934 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 >> 172.21.15.63:0/1483305000 conn(0x5629f7545f00 msgr2=0x5629f761e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 shutdown_connections 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.826+0000 7f6c2cb7fb80 1 -- 172.21.15.63:0/1483305000 wait complete. 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c2cb7fb80 20 client.24934 trim_cache size 0 max 16384 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 0, 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24533, 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 177.93176890300001, 2024-04-25T18:45:30.836 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 201.84813559700001 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 msgr2=0x7f6c000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 0x7f6c000a6100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c1111b0 tx=0x7f6c08009260 comp rx=0 tx=0).stop 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 msgr2=0x7f6c1c19b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0057f0 tx=0x7f6c0c00bfd0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 shutdown_connections 2024-04-25T18:45:30.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f6c000a3c40 0x7f6c000a6100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6c1c10f4b0 0x7f6c1c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c1c074c70 0x7f6c1c19b390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 --2- 172.21.15.63:0/2522775163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c1c075590 0x7f6c1c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.827+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 >> 172.21.15.63:0/2522775163 conn(0x7f6c1c079c00 msgr2=0x7f6c1c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.828+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 shutdown_connections 2024-04-25T18:45:30.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:30.828+0000 7f6c22200700 1 -- 172.21.15.63:0/2522775163 wait complete. 2024-04-25T18:45:31.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:30 smithi063 ceph-mon[28006]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:31.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:30 smithi070 ceph-mon[32820]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:31.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:30 smithi114 ceph-mon[32988]: pgmap v434: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:31.651 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T18:45:31.651 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T18:45:31.660 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:45:31.660 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T18:45:32.141 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:33.132 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:32 smithi063 ceph-mon[28006]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:33.245 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:32 smithi070 ceph-mon[32820]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:33.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:32 smithi114 ceph-mon[32988]: pgmap v435: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 -- 172.21.15.63:0/2146486702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410a2f0 msgr2=0x7f1c3410a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/2146486702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410a2f0 0x7f1c3410a750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009990 tx=0x7f1c20009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 -- 172.21.15.63:0/2146486702 shutdown_connections 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/2146486702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c3410ac90 0x7f1c34111540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/2146486702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c341090f0 0x7f1c34109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/2146486702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410a2f0 0x7f1c3410a750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 -- 172.21.15.63:0/2146486702 >> 172.21.15.63:0/2146486702 conn(0x7f1c340780c0 msgr2=0x7f1c3407a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 -- 172.21.15.63:0/2146486702 shutdown_connections 2024-04-25T18:45:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.392+0000 7f1c3ba00700 1 -- 172.21.15.63:0/2146486702 wait complete. 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.394+0000 7f1c3ba00700 1 Processor -- start 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.394+0000 7f1c3ba00700 1 -- start start 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.394+0000 7f1c3ba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.394+0000 7f1c3ba00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 0x7f1c3419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.394+0000 7f1c3ba00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410ac90 0x7f1c341a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c3ba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34114bc0 con 0x7f1c341090f0 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c3ba00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34114a40 con 0x7f1c3410a2f0 2024-04-25T18:45:34.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c3ba00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1c34114d40 con 0x7f1c3410ac90 2024-04-25T18:45:34.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c39000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:34.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c39000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45510/0 (socket says 172.21.15.63:45510) 2024-04-25T18:45:34.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c39000700 1 -- 172.21.15.63:0/3260808633 learned_addr learned my addr 172.21.15.63:0/3260808633 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:34.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 0x7f1c3419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:34.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c39a00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410ac90 0x7f1c341a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 -- 172.21.15.63:0/3260808633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410ac90 msgr2=0x7f1c341a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410ac90 0x7f1c341a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 -- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 msgr2=0x7f1c3419f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c2be00700 1 -- 172.21.15.63:0/3260808633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c341a6ea0 con 0x7f1c3410a2f0 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.395+0000 7f1c39000700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:34.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c2be00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 0x7f1c3419f7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000b850 tx=0x7f1c20009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20026070 con 0x7f1c3410a2f0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c20021410 con 0x7f1c3410a2f0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20017650 con 0x7f1c3410a2f0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c341a7130 con 0x7f1c3410a2f0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.396+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c341a75e0 con 0x7f1c3410a2f0 2024-04-25T18:45:34.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.397+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c34069730 con 0x7f1c3410a2f0 2024-04-25T18:45:34.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.399+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1c20029070 con 0x7f1c3410a2f0 2024-04-25T18:45:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.401+0000 7f1c32000700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 0x7f1c100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.401+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f1c200d11c0 con 0x7f1c3410a2f0 2024-04-25T18:45:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.401+0000 7f1c39000700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 0x7f1c100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:34.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.402+0000 7f1c39000700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 0x7f1c100a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c18005950 tx=0x7f1c18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:34.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.405+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c2005dff0 con 0x7f1c3410a2f0 2024-04-25T18:45:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.655+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f1c34062e80 con 0x7f1c100a3cc0 2024-04-25T18:45:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.657+0000 7f1c32000700 1 -- 172.21.15.63:0/3260808633 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1c34062e80 con 0x7f1c100a3cc0 2024-04-25T18:45:34.659 INFO:teuthology.orchestra.run.smithi063.stdout:Backend: cephadm 2024-04-25T18:45:34.659 INFO:teuthology.orchestra.run.smithi063.stdout:Available: Yes 2024-04-25T18:45:34.659 INFO:teuthology.orchestra.run.smithi063.stdout:Paused: No 2024-04-25T18:45:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 msgr2=0x7f1c100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 0x7f1c100a6180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1c18005950 tx=0x7f1c18009260 comp rx=0 tx=0).stop 2024-04-25T18:45:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 msgr2=0x7f1c3419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 0x7f1c3419f7d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000b850 tx=0x7f1c20009580 comp rx=0 tx=0).stop 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 shutdown_connections 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f1c100a3cc0 0x7f1c100a6180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1c3410ac90 0x7f1c341a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1c341090f0 0x7f1c3419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 --2- 172.21.15.63:0/3260808633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c3410a2f0 0x7f1c3419f7d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 >> 172.21.15.63:0/3260808633 conn(0x7f1c340780c0 msgr2=0x7f1c3410d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.660+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 shutdown_connections 2024-04-25T18:45:34.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:34.661+0000 7f1c3ba00700 1 -- 172.21.15.63:0/3260808633 wait complete. 2024-04-25T18:45:35.097 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:34 smithi063 ceph-mon[28006]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:34 smithi070 ceph-mon[32820]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:34 smithi114 ceph-mon[32988]: pgmap v436: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:35.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T18:45:36.001 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:36.150 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:35 smithi063 ceph-mon[28006]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:35 smithi070 ceph-mon[32820]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:35 smithi114 ceph-mon[32988]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:37.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:36 smithi070 ceph-mon[32820]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:37.206 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:36 smithi063 ceph-mon[28006]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:37.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:36 smithi114 ceph-mon[32988]: pgmap v437: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 -- 172.21.15.63:0/3165968310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 msgr2=0x7fc5b410a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 --2- 172.21.15.63:0/3165968310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b410a740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009920 tx=0x7fc5a4009c30 comp rx=0 tx=0).stop 2024-04-25T18:45:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 -- 172.21.15.63:0/3165968310 shutdown_connections 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 --2- 172.21.15.63:0/3165968310 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b410ac80 0x7fc5b4111530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 --2- 172.21.15.63:0/3165968310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b41090e0 0x7fc5b4109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 --2- 172.21.15.63:0/3165968310 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b410a740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 -- 172.21.15.63:0/3165968310 >> 172.21.15.63:0/3165968310 conn(0x7fc5b40780b0 msgr2=0x7fc5b407a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 -- 172.21.15.63:0/3165968310 shutdown_connections 2024-04-25T18:45:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.194+0000 7fc5bc800700 1 -- 172.21.15.63:0/3165968310 wait complete. 2024-04-25T18:45:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.195+0000 7fc5bc800700 1 Processor -- start 2024-04-25T18:45:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.195+0000 7fc5bc800700 1 -- start start 2024-04-25T18:45:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b41090e0 0x7fc5b419f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b410ac80 0x7fc5b41a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b4114bb0 con 0x7fc5b410ac80 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b4114a30 con 0x7fc5b410a2e0 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5bc800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b4114d30 con 0x7fc5b41090e0 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5b9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5b9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53262/0 (socket says 172.21.15.63:53262) 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5b9400700 1 -- 172.21.15.63:0/1438811623 learned_addr learned my addr 172.21.15.63:0/1438811623 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5ba800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b410ac80 0x7fc5b41a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.196+0000 7fc5b9e00700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b41090e0 0x7fc5b419f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 -- 172.21.15.63:0/1438811623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b41090e0 msgr2=0x7fc5b419f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b41090e0 0x7fc5b419f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 -- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b410ac80 msgr2=0x7fc5b41a6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b410ac80 0x7fc5b41a6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 -- 172.21.15.63:0/1438811623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b41a6f10 con 0x7fc5b410a2e0 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5b9400700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009840 tx=0x7fc5a4005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.197+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4025070 con 0x7fc5b410a2e0 2024-04-25T18:45:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.198+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5a4005670 con 0x7fc5b410a2e0 2024-04-25T18:45:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.198+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b41a71a0 con 0x7fc5b410a2e0 2024-04-25T18:45:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.198+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4003b50 con 0x7fc5b410a2e0 2024-04-25T18:45:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.198+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b41a75f0 con 0x7fc5b410a2e0 2024-04-25T18:45:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.199+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b4069730 con 0x7fc5b410a2e0 2024-04-25T18:45:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.201+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc5a4028070 con 0x7fc5b410a2e0 2024-04-25T18:45:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.202+0000 7fc5aea00700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 0x7fc59c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.203+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fc5a40d0c40 con 0x7fc5b410a2e0 2024-04-25T18:45:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.203+0000 7fc5b9e00700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 0x7fc59c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.203+0000 7fc5b9e00700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 0x7fc59c0a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc5b410a140 tx=0x7fc5a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:38.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.205+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5a4013030 con 0x7fc5b410a2e0 2024-04-25T18:45:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.444+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fc5b4062e80 con 0x7fc59c0a3cc0 2024-04-25T18:45:38.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.486+0000 7fc5aea00700 1 -- 172.21.15.63:0/1438811623 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fc5b4062e80 con 0x7fc59c0a3cc0 2024-04-25T18:45:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T18:45:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:mds.a smithi063 running (3m) 42s ago 3m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 1323d794ac39 2024-04-25T18:45:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:mds.b smithi070 running (3m) 71s ago 3m 30.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 b488c0f6f4b0 2024-04-25T18:45:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:mds.c smithi114 running (3m) 71s ago 3m 19.6M - 19.0.0-3156-gd69f13d8 da4b004c09e8 865e95a7b809 2024-04-25T18:45:38.490 INFO:teuthology.orchestra.run.smithi063.stdout:mds.d smithi063 running (3m) 42s ago 3m 13.3M - 19.0.0-3156-gd69f13d8 da4b004c09e8 0dd4e385d17b 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.e smithi070 running (3m) 71s ago 3m 21.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 27594d2271bb 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.f smithi114 running (3m) 71s ago 3m 22.4M - 19.0.0-3156-gd69f13d8 da4b004c09e8 c2ebf294e58c 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.g smithi063 running (3m) 42s ago 3m 13.0M - 19.0.0-3156-gd69f13d8 da4b004c09e8 750410dee27f 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.h smithi070 running (3m) 71s ago 3m 18.7M - 19.0.0-3156-gd69f13d8 da4b004c09e8 652fc676a66e 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.i smithi114 running (3m) 71s ago 3m 21.9M - 19.0.0-3156-gd69f13d8 da4b004c09e8 2a07dcf4c58e 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.j smithi063 running (4m) 42s ago 4m 13.2M - 19.0.0-3156-gd69f13d8 da4b004c09e8 6619c2d134c6 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.k smithi070 running (4m) 71s ago 4m 14.8M - 19.0.0-3156-gd69f13d8 da4b004c09e8 551e3f29509b 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mds.l smithi114 running (4m) 71s ago 4m 14.6M - 19.0.0-3156-gd69f13d8 da4b004c09e8 530e75fe5427 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.x smithi063 *:9283,8765 running (12m) 42s ago 12m 516M - 19.0.0-3156-gd69f13d8 da4b004c09e8 fa158b16e4c8 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.y smithi070 *:8443,8765 running (10m) 71s ago 10m 453M - 19.0.0-3156-gd69f13d8 da4b004c09e8 d2e1b3ad2a33 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mgr.z smithi114 *:8443,8765 running (10m) 71s ago 10m 454M - 19.0.0-3156-gd69f13d8 da4b004c09e8 cb781d44ef4d 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mon.a smithi063 running (12m) 42s ago 12m 50.6M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 329dd1f8f9f9 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mon.b smithi070 running (10m) 71s ago 10m 43.6M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 229d68a239fe 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:mon.c smithi114 running (10m) 71s ago 10m 42.5M 2048M 19.0.0-3156-gd69f13d8 da4b004c09e8 e4318e791594 2024-04-25T18:45:38.491 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 smithi063 running (9m) 42s ago 9m 56.3M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 2fa3e6f40e2a 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 smithi070 running (9m) 71s ago 9m 53.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 881380d11f46 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 smithi114 running (8m) 71s ago 8m 51.0M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 34258b0e80ac 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 smithi063 running (8m) 42s ago 8m 54.2M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 490cf821db8f 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 smithi070 running (7m) 71s ago 7m 46.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 954fafa52a0f 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 smithi114 running (7m) 71s ago 7m 56.8M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 3c25df63a4af 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6 smithi063 running (6m) 42s ago 6m 44.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 bf42b622a0fd 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7 smithi070 running (6m) 71s ago 6m 56.5M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 599dd8e7cee7 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.8 smithi114 running (6m) 71s ago 6m 56.4M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 2c1a14a52029 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.9 smithi063 running (5m) 42s ago 5m 50.8M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 6fa27f8ed288 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.10 smithi070 running (5m) 71s ago 5m 55.9M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 9b2b2271385b 2024-04-25T18:45:38.492 INFO:teuthology.orchestra.run.smithi063.stdout:osd.11 smithi114 running (4m) 71s ago 4m 55.7M 4272M 19.0.0-3156-gd69f13d8 da4b004c09e8 88ea9569afac 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 msgr2=0x7fc59c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 0x7fc59c0a6180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc5b410a140 tx=0x7fc5a8009260 comp rx=0 tx=0).stop 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 msgr2=0x7fc5b419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009840 tx=0x7fc5a4005270 comp rx=0 tx=0).stop 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 shutdown_connections 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc59c0a3cc0 0x7fc59c0a6180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc5b41090e0 0x7fc5b419f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5b410ac80 0x7fc5b41a6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 --2- 172.21.15.63:0/1438811623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5b410a2e0 0x7fc5b419f7f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:38.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 >> 172.21.15.63:0/1438811623 conn(0x7fc5b40780b0 msgr2=0x7fc5b410d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:38.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 shutdown_connections 2024-04-25T18:45:38.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:38.490+0000 7fc5bc800700 1 -- 172.21.15.63:0/1438811623 wait complete. 2024-04-25T18:45:39.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:38 smithi063 ceph-mon[28006]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:39.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:38 smithi070 ceph-mon[32820]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:39.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:38 smithi114 ceph-mon[32988]: pgmap v438: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:39.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T18:45:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:40.184 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:39 smithi063 ceph-mon[28006]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:40.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:39 smithi070 ceph-mon[32820]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:39 smithi114 ceph-mon[32988]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:41.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:40 smithi070 ceph-mon[32820]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:41.218 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:40 smithi114 ceph-mon[32988]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:41.233 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:40 smithi063 ceph-mon[28006]: pgmap v439: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:42.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 -- 172.21.15.63:0/3250916643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 msgr2=0x7fbf4c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:42.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 --2- 172.21.15.63:0/3250916643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c102100 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbf380098f0 tx=0x7fbf38009c00 comp rx=0 tx=0).stop 2024-04-25T18:45:42.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 -- 172.21.15.63:0/3250916643 shutdown_connections 2024-04-25T18:45:42.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 --2- 172.21.15.63:0/3250916643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 0x7fbf4c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 --2- 172.21.15.63:0/3250916643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 --2- 172.21.15.63:0/3250916643 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c102100 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 -- 172.21.15.63:0/3250916643 >> 172.21.15.63:0/3250916643 conn(0x7fbf4c0fcf60 msgr2=0x7fbf4c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:42.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 -- 172.21.15.63:0/3250916643 shutdown_connections 2024-04-25T18:45:42.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.127+0000 7fbf52600700 1 -- 172.21.15.63:0/3250916643 wait complete. 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.128+0000 7fbf52600700 1 Processor -- start 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 -- start start 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 0x7fbf4c1a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c110770 con 0x7fbf4c101320 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c1105f0 con 0x7fbf4c102640 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf52600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c1108f0 con 0x7fbf4c101c80 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:42.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4b400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40690/0 (socket says 172.21.15.63:40690) 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42856/0 (socket says 172.21.15.63:42856) 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4b400700 1 -- 172.21.15.63:0/2424790323 learned_addr learned my addr 172.21.15.63:0/2424790323 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.129+0000 7fbf4be00700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 0x7fbf4c1a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 -- 172.21.15.63:0/2424790323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 msgr2=0x7fbf4c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 -- 172.21.15.63:0/2424790323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 msgr2=0x7fbf4c1a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 0x7fbf4c1a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 -- 172.21.15.63:0/2424790323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4c1a2af0 con 0x7fbf4c101320 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf4b400700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00ec40 tx=0x7fbf3c00ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:42.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.130+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c00ccf0 con 0x7fbf4c101320 2024-04-25T18:45:42.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.131+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4c1a2de0 con 0x7fbf4c101320 2024-04-25T18:45:42.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.131+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf3c00ce90 con 0x7fbf4c101320 2024-04-25T18:45:42.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.131+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c010430 con 0x7fbf4c101320 2024-04-25T18:45:42.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.131+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4c1a3240 con 0x7fbf4c101320 2024-04-25T18:45:42.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.132+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4c069730 con 0x7fbf4c101320 2024-04-25T18:45:42.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.134+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbf3c004000 con 0x7fbf4c101320 2024-04-25T18:45:42.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.135+0000 7fbf43e00700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 0x7fbf300a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:42.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.135+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fbf3c0d06c0 con 0x7fbf4c101320 2024-04-25T18:45:42.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.136+0000 7fbf4aa00700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 0x7fbf300a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:42.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.136+0000 7fbf4aa00700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 0x7fbf300a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800faf0 tx=0x7fbf38019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:42.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.141+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf3c09d010 con 0x7fbf4c101320 2024-04-25T18:45:42.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.383+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fbf4c062e80 con 0x7fbf300a3cc0 2024-04-25T18:45:42.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.400+0000 7fbf43e00700 1 -- 172.21.15.63:0/2424790323 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fbf4c062e80 con 0x7fbf300a3cc0 2024-04-25T18:45:42.403 INFO:teuthology.orchestra.run.smithi063.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T18:45:42.403 INFO:teuthology.orchestra.run.smithi063.stdout:mds.all 12/12 75s ago 4m smithi063=a;smithi063=d;smithi063=g;smithi063=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi114=c;smithi114=f;smithi114=i;smithi114=l;count:12 2024-04-25T18:45:42.403 INFO:teuthology.orchestra.run.smithi063.stdout:mgr 3/3 75s ago 10m smithi063=x;smithi070=y;smithi114=z;count:3 2024-04-25T18:45:42.403 INFO:teuthology.orchestra.run.smithi063.stdout:mon 3/3 75s ago 10m smithi063:172.21.15.63=a;smithi070:172.21.15.70=b;smithi114:172.21.15.114=c;count:3 2024-04-25T18:45:42.403 INFO:teuthology.orchestra.run.smithi063.stdout:osd 12 75s ago - 2024-04-25T18:45:42.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.403+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 msgr2=0x7fbf300a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.403+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 0x7fbf300a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800faf0 tx=0x7fbf38019040 comp rx=0 tx=0).stop 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.403+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 msgr2=0x7fbf4c19aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.403+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00ec40 tx=0x7fbf3c00ef50 comp rx=0 tx=0).stop 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 shutdown_connections 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fbf300a3cc0 0x7fbf300a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf4c102640 0x7fbf4c1a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf4c101320 0x7fbf4c19aee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 --2- 172.21.15.63:0/2424790323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf4c101c80 0x7fbf4c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 >> 172.21.15.63:0/2424790323 conn(0x7fbf4c0fcf60 msgr2=0x7fbf4c0fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 shutdown_connections 2024-04-25T18:45:42.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:42.404+0000 7fbf52600700 1 -- 172.21.15.63:0/2424790323 wait complete. 2024-04-25T18:45:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:42 smithi063 ceph-mon[28006]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:43.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:42 smithi070 ceph-mon[32820]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:43.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T18:45:43.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:42 smithi114 ceph-mon[32988]: pgmap v440: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:43.785 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:44.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:43 smithi070 ceph-mon[32820]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:44.208 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:43 smithi063 ceph-mon[28006]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:44.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:43 smithi114 ceph-mon[32988]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:45.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:44 smithi063 ceph-mon[28006]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:45.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:44 smithi070 ceph-mon[32820]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:45.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:44 smithi114 ceph-mon[32988]: pgmap v441: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 -- 172.21.15.63:0/1267782402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618105ee0 msgr2=0x7fd618106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 --2- 172.21.15.63:0/1267782402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618105ee0 0x7fd618106340 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd6040099c0 tx=0x7fd604009cd0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 -- 172.21.15.63:0/1267782402 shutdown_connections 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 --2- 172.21.15.63:0/1267782402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618104ce0 0x7fd618105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 --2- 172.21.15.63:0/1267782402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618106880 0x7fd61810d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 --2- 172.21.15.63:0/1267782402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618105ee0 0x7fd618106340 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 -- 172.21.15.63:0/1267782402 >> 172.21.15.63:0/1267782402 conn(0x7fd6181002a0 msgr2=0x7fd6181026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 -- 172.21.15.63:0/1267782402 shutdown_connections 2024-04-25T18:45:46.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.011+0000 7fd61ee00700 1 -- 172.21.15.63:0/1267782402 wait complete. 2024-04-25T18:45:46.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.012+0000 7fd61ee00700 1 Processor -- start 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 -- start start 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 0x7fd61819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 0x7fd61819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 0x7fd6181a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181107a0 con 0x7fd618104ce0 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd618110620 con 0x7fd618105ee0 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ee00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd618110920 con 0x7fd618106880 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd617e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 0x7fd61819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd617e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 0x7fd61819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40718/0 (socket says 172.21.15.63:40718) 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd617e00700 1 -- 172.21.15.63:0/3871421688 learned_addr learned my addr 172.21.15.63:0/3871421688 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd617400700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 0x7fd61819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:46.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.013+0000 7fd61ce00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 0x7fd6181a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:46.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 -- 172.21.15.63:0/3871421688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 msgr2=0x7fd6181a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 0x7fd6181a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 -- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 msgr2=0x7fd61819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 0x7fd61819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 -- 172.21.15.63:0/3871421688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6181a2a40 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd61ce00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 0x7fd6181a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd617400700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 0x7fd61819b320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd6040098e0 tx=0x7fd604003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60400bea0 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd60400f460 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604017670 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.014+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6181a2cd0 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.015+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6181a3120 con 0x7fd618105ee0 2024-04-25T18:45:46.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.016+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd61802df80 con 0x7fd618105ee0 2024-04-25T18:45:46.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.018+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd60400f8a0 con 0x7fd618105ee0 2024-04-25T18:45:46.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.019+0000 7fd614c00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 0x7fd5fc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:46.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.019+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fd6040d11f0 con 0x7fd618105ee0 2024-04-25T18:45:46.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.019+0000 7fd617e00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 0x7fd5fc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:46.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.020+0000 7fd617e00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 0x7fd5fc0a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd60800bd70 tx=0x7fd60800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:46.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.024+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd604013070 con 0x7fd618105ee0 2024-04-25T18:45:46.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.265+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fd618062e80 con 0x7fd5fc0a3c40 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.268+0000 7fd614c00700 1 -- 172.21.15.63:0/3871421688 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fd618062e80 con 0x7fd5fc0a3c40 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:HOST ADDR LABELS STATUS 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063 172.21.15.63 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:smithi070 172.21.15.70 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:smithi114 172.21.15.114 2024-04-25T18:45:46.270 INFO:teuthology.orchestra.run.smithi063.stdout:3 hosts in cluster 2024-04-25T18:45:46.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 msgr2=0x7fd5fc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:46.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 0x7fd5fc0a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd60800bd70 tx=0x7fd60800b4a0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 msgr2=0x7fd61819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:46.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 0x7fd61819b320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd6040098e0 tx=0x7fd604003e60 comp rx=0 tx=0).stop 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 shutdown_connections 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fd5fc0a3c40 0x7fd5fc0a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd618106880 0x7fd6181a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd618104ce0 0x7fd61819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 --2- 172.21.15.63:0/3871421688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd618105ee0 0x7fd61819b320 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 >> 172.21.15.63:0/3871421688 conn(0x7fd6181002a0 msgr2=0x7fd6181091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.271+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 shutdown_connections 2024-04-25T18:45:46.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:46.272+0000 7fd61ee00700 1 -- 172.21.15.63:0/3871421688 wait complete. 2024-04-25T18:45:47.048 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:46 smithi063 ceph-mon[28006]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:47.129 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T18:45:47.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:46 smithi070 ceph-mon[32820]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:47.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:46 smithi114 ceph-mon[32988]: pgmap v442: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:47.601 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:48.043 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:47 smithi063 ceph-mon[28006]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:48.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:47 smithi114 ceph-mon[32988]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:48.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:47 smithi070 ceph-mon[32820]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:49.048 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:48 smithi063 ceph-mon[28006]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:49.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:48 smithi114 ceph-mon[32988]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:49.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:48 smithi070 ceph-mon[32820]: pgmap v443: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 -- 172.21.15.63:0/2059028523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 msgr2=0x7f3e74077810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 --2- 172.21.15.63:0/2059028523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e74077810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009990 tx=0x7f3e64009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 -- 172.21.15.63:0/2059028523 shutdown_connections 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 --2- 172.21.15.63:0/2059028523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e74077810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 --2- 172.21.15.63:0/2059028523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74079000 0x7f3e74079880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 --2- 172.21.15.63:0/2059028523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74078640 0x7f3e74078ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 -- 172.21.15.63:0/2059028523 >> 172.21.15.63:0/2059028523 conn(0x7f3e74100290 msgr2=0x7f3e741026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 -- 172.21.15.63:0/2059028523 shutdown_connections 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.912+0000 7f3e7b000700 1 -- 172.21.15.63:0/2059028523 wait complete. 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.913+0000 7f3e7b000700 1 Processor -- start 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 -- start start 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:49.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74078640 0x7f3e7419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:49.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 0x7f3e741a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:49.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74114bb0 con 0x7f3e74078640 2024-04-25T18:45:49.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74114a30 con 0x7f3e74079000 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e7b000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74114d30 con 0x7f3e740773f0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.914+0000 7f3e73e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46854/0 (socket says 172.21.15.63:46854) 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 -- 172.21.15.63:0/1058339667 learned_addr learned my addr 172.21.15.63:0/1058339667 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73400700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74078640 0x7f3e7419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e79000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 0x7f3e741a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 -- 172.21.15.63:0/1058339667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 msgr2=0x7f3e741a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 0x7f3e741a67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 -- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74078640 msgr2=0x7f3e7419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74078640 0x7f3e7419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 -- 172.21.15.63:0/1058339667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e741a6e50 con 0x7f3e740773f0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e79000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 0x7f3e741a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.915+0000 7f3e73e00700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3e64000c00 tx=0x7f3e640047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:49.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.916+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64027070 con 0x7f3e740773f0 2024-04-25T18:45:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.916+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e741a70e0 con 0x7f3e740773f0 2024-04-25T18:45:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.916+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e64022470 con 0x7f3e740773f0 2024-04-25T18:45:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.916+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e64017650 con 0x7f3e740773f0 2024-04-25T18:45:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.916+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e741a7530 con 0x7f3e740773f0 2024-04-25T18:45:49.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.917+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e74069730 con 0x7f3e740773f0 2024-04-25T18:45:49.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.919+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3e64009d10 con 0x7f3e740773f0 2024-04-25T18:45:49.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.920+0000 7f3e70c00700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 0x7f3e580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:49.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.921+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f3e640d11c0 con 0x7f3e740773f0 2024-04-25T18:45:49.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.921+0000 7f3e73400700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 0x7f3e580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:49.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.921+0000 7f3e73400700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 0x7f3e580a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009c60 tx=0x7f3e60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:49.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:49.923+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e64013070 con 0x7f3e740773f0 2024-04-25T18:45:50.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.178+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 --> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3e74062e80 con 0x7f3e580a3c40 2024-04-25T18:45:50.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.181+0000 7f3e70c00700 1 -- 172.21.15.63:0/1058339667 <== mgr.14152 v2:172.21.15.63:6800/3659595029 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3e74062e80 con 0x7f3e580a3c40 2024-04-25T18:45:50.183 INFO:teuthology.orchestra.run.smithi063.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T18:45:50.183 INFO:teuthology.orchestra.run.smithi063.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-04-25T18:45:50.183 INFO:teuthology.orchestra.run.smithi063.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 5m ago Has a FileSystem, LVM detected 2024-04-25T18:45:50.183 INFO:teuthology.orchestra.run.smithi063.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 msgr2=0x7f3e580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 0x7f3e580a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009c60 tx=0x7f3e60009260 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 msgr2=0x7f3e7419f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3e64000c00 tx=0x7f3e640047b0 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 shutdown_connections 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3e580a3c40 0x7f3e580a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3e740773f0 0x7f3e7419f1f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e74079000 0x7f3e741a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 --2- 172.21.15.63:0/1058339667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e74078640 0x7f3e7419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 >> 172.21.15.63:0/1058339667 conn(0x7f3e74100290 msgr2=0x7f3e74101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:50.187 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.184+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 shutdown_connections 2024-04-25T18:45:50.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:50.185+0000 7f3e7b000700 1 -- 172.21.15.63:0/1058339667 wait complete. 2024-04-25T18:45:50.981 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T18:45:50.991 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:45:50.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T18:45:51.255 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:50 smithi070 ceph-mon[32820]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:51.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:50 smithi114 ceph-mon[32988]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:51.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:50 smithi063 ceph-mon[28006]: pgmap v444: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:45:51.480 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:51 smithi114 ceph-mon[32988]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:52.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:51 smithi063 ceph-mon[28006]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:52.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:51 smithi070 ceph-mon[32820]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:45:53.228 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:52 smithi063 ceph-mon[28006]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:53.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:52 smithi114 ceph-mon[32988]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:53.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:52 smithi070 ceph-mon[32820]: pgmap v445: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 -- 172.21.15.63:0/3719973261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe998076a70 msgr2=0x7fe998074ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 --2- 172.21.15.63:0/3719973261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe998076a70 0x7fe998074ed0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe988009990 tx=0x7fe988009ca0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 -- 172.21.15.63:0/3719973261 shutdown_connections 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 --2- 172.21.15.63:0/3719973261 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe99810a990 0x7fe99810ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 --2- 172.21.15.63:0/3719973261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe998076a70 0x7fe998074ed0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 --2- 172.21.15.63:0/3719973261 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe9980754a0 0x7fe998075920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.742+0000 7fe99ec00700 1 -- 172.21.15.63:0/3719973261 >> 172.21.15.63:0/3719973261 conn(0x7fe9980fe060 msgr2=0x7fe9981004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.743+0000 7fe99ec00700 1 -- 172.21.15.63:0/3719973261 shutdown_connections 2024-04-25T18:45:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.743+0000 7fe99ec00700 1 -- 172.21.15.63:0/3719973261 wait complete. 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 Processor -- start 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 -- start start 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 0x7fe99819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe99810a990 0x7fe9981a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe998110730 con 0x7fe99810a990 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981105b0 con 0x7fe9980754a0 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.744+0000 7fe99ec00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe9981108b0 con 0x7fe998076a70 2024-04-25T18:45:53.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe99cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe99810a990 0x7fe9981a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:37270/0 (socket says 172.21.15.63:37270) 2024-04-25T18:45:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 -- 172.21.15.63:0/2363807415 learned_addr learned my addr 172.21.15.63:0/2363807415 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 0x7fe99819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 -- 172.21.15.63:0/2363807415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 msgr2=0x7fe99819b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 0x7fe99819b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 -- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe99810a990 msgr2=0x7fe9981a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe99810a990 0x7fe9981a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997e00700 1 -- 172.21.15.63:0/2363807415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9981a2a40 con 0x7fe9980754a0 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.745+0000 7fe997400700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 0x7fe99819b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe997e00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9880059d0 tx=0x7fe9880047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988027070 con 0x7fe9980754a0 2024-04-25T18:45:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9981a2cd0 con 0x7fe9980754a0 2024-04-25T18:45:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe98800fbc0 con 0x7fe9980754a0 2024-04-25T18:45:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880173f0 con 0x7fe9980754a0 2024-04-25T18:45:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.746+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9981a3180 con 0x7fe9980754a0 2024-04-25T18:45:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.747+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe998069730 con 0x7fe9980754a0 2024-04-25T18:45:53.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.749+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe98802a070 con 0x7fe9980754a0 2024-04-25T18:45:53.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.751+0000 7fe994c00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 0x7fe97c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:53.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.751+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fe9880d1740 con 0x7fe9980754a0 2024-04-25T18:45:53.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.751+0000 7fe997400700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 0x7fe97c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:53.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.752+0000 7fe997400700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 0x7fe97c0a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe9980767a0 tx=0x7fe984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:53.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:53.753+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe988013070 con 0x7fe9980754a0 2024-04-25T18:45:54.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.038+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fe99802df80 con 0x7fe9980754a0 2024-04-25T18:45:54.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.039+0000 7fe994c00700 1 -- 172.21.15.63:0/2363807415 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3048 (secure 0 0 0) 0x7fe98809e120 con 0x7fe9980754a0 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:e23 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:legacy client fscid: 1 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:Filesystem 'cephfs' (1) 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:fs_name cephfs 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 23 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-04-25T18:42:31.925526+0000 2024-04-25T18:45:54.043 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-04-25T18:42:57.440972+0000 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:tableserver 0 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:root 0 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:session_timeout 300 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:session_autoclose 300 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:max_file_size 1099511627776 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:max_xattr_size 65536 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:required_client_features {} 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure 0 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:last_failure_osd_epoch 0 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:max_mds 3 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:in 0,1,2 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:up {0=24533,1=24520,2=24508} 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:failed 2024-04-25T18:45:54.044 INFO:teuthology.orchestra.run.smithi063.stdout:damaged 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:stopped 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:data_pools [3] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:metadata_pool 2 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:inline_data disabled 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:balancer 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:bal_rank_mask -1 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:standby_count_wanted 0 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:qdb_cluster leader: 24533 members: 24533,24508,24520 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.b{0:24533} state up:active seq 7 addr [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.f{0:24496} state up:standby-replay seq 1 addr [v2:172.21.15.114:6834/3172988525,v1:172.21.15.114:6835/3172988525] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.i{1:24520} state up:active seq 15 addr [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.h{1:24494} state up:standby-replay seq 1 addr [v2:172.21.15.70:6836/4043589496,v1:172.21.15.70:6837/4043589496] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.c{2:24508} state up:active seq 16 addr [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.045 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.e{2:24473} state up:standby-replay seq 1 addr [v2:172.21.15.70:6834/3182056159,v1:172.21.15.70:6835/3182056159] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:Standby daemons: 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.g{-1:14391} state up:standby seq 1 addr [v2:172.21.15.63:6838/3937073235,v1:172.21.15.63:6839/3937073235] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.a{-1:14403} state up:standby seq 1 addr [v2:172.21.15.63:6840/512888768,v1:172.21.15.63:6841/512888768] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.k{-1:24443} state up:standby seq 1 addr [v2:172.21.15.70:6832/2179103180,v1:172.21.15.70:6833/2179103180] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.114:6832/1215662020,v1:172.21.15.114:6833/1215662020] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.j{-1:24455} state up:standby seq 1 addr [v2:172.21.15.63:6834/3714473913,v1:172.21.15.63:6835/3714473913] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.046 INFO:teuthology.orchestra.run.smithi063.stdout:[mds.d{-1:24461} state up:standby seq 1 addr [v2:172.21.15.63:6836/166123931,v1:172.21.15.63:6837/166123931] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 msgr2=0x7fe97c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 0x7fe97c0a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe9980767a0 tx=0x7fe984009260 comp rx=0 tx=0).stop 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 msgr2=0x7fe99819ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe9880059d0 tx=0x7fe9880047b0 comp rx=0 tx=0).stop 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 shutdown_connections 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fe97c0a3c40 0x7fe97c0a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe99810a990 0x7fe9981a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9980754a0 0x7fe99819ae30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 --2- 172.21.15.63:0/2363807415 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe998076a70 0x7fe99819b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 >> 172.21.15.63:0/2363807415 conn(0x7fe9980fe060 msgr2=0x7fe998106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 shutdown_connections 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:54.043+0000 7fe99ec00700 1 -- 172.21.15.63:0/2363807415 wait complete. 2024-04-25T18:45:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:45:54.922 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T18:45:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:54 smithi063 ceph-mon[28006]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:54 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2363807415' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T18:45:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:54 smithi114 ceph-mon[32988]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:55.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:54 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2363807415' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T18:45:55.395 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:55.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:54 smithi070 ceph-mon[32820]: pgmap v446: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:45:55.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:54 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2363807415' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T18:45:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:56 smithi063 ceph-mon[28006]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:57.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:56 smithi114 ceph-mon[32988]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:57.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:56 smithi070 ceph-mon[32820]: pgmap v447: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:45:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 -- 172.21.15.63:0/1923581930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101c80 msgr2=0x7f36f4102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 --2- 172.21.15.63:0/1923581930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101c80 0x7f36f4102100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0098f0 tx=0x7f36dc009c00 comp rx=0 tx=0).stop 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 -- 172.21.15.63:0/1923581930 shutdown_connections 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 --2- 172.21.15.63:0/1923581930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 0x7f36f4102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 --2- 172.21.15.63:0/1923581930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101320 0x7f36f4101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 --2- 172.21.15.63:0/1923581930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101c80 0x7f36f4102100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 -- 172.21.15.63:0/1923581930 >> 172.21.15.63:0/1923581930 conn(0x7f36f40fcf40 msgr2=0x7f36f40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 -- 172.21.15.63:0/1923581930 shutdown_connections 2024-04-25T18:45:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.704+0000 7f36fa000700 1 -- 172.21.15.63:0/1923581930 wait complete. 2024-04-25T18:45:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 Processor -- start 2024-04-25T18:45:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 -- start start 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101c80 0x7f36f419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 0x7f36f41a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f36f4110770 con 0x7f36f4101c80 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36f41105f0 con 0x7f36f4102640 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36fa000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f36f41108f0 con 0x7f36f4101320 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36f3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36f3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50814/0 (socket says 172.21.15.63:50814) 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36f3400700 1 -- 172.21.15.63:0/3404760708 learned_addr learned my addr 172.21.15.63:0/3404760708 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.706+0000 7f36ebe00700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101c80 0x7f36f419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3e00700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 0x7f36f41a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 -- 172.21.15.63:0/3404760708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 msgr2=0x7f36f41a2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 0x7f36f41a2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 -- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101c80 msgr2=0x7f36f419b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101c80 0x7f36f419b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 -- 172.21.15.63:0/3404760708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f41a2b30 con 0x7f36f4101320 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.707+0000 7f36f3400700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36e400ec40 tx=0x7f36e400ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.708+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e400ccf0 con 0x7f36f4101320 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.708+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f41a2e20 con 0x7f36f4101320 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.708+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36e400ce90 con 0x7f36f4101320 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.708+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4010430 con 0x7f36f4101320 2024-04-25T18:45:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.708+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f41a3280 con 0x7f36f4101320 2024-04-25T18:45:57.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.709+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f4069730 con 0x7f36f4101320 2024-04-25T18:45:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.711+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f36e4004000 con 0x7f36f4101320 2024-04-25T18:45:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.713+0000 7f36f0c00700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 0x7f36d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:45:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.713+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f36e40d0770 con 0x7f36f4101320 2024-04-25T18:45:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.713+0000 7f36ebe00700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 0x7f36d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:45:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.714+0000 7f36ebe00700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 0x7f36d80a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009810 tx=0x7f36dc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:45:57.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.717+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36e409d0c0 con 0x7f36f4101320 2024-04-25T18:45:57.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.951+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f36f4063650 con 0x7f36f4101320 2024-04-25T18:45:57.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.953+0000 7f36f0c00700 1 -- 172.21.15.63:0/3404760708 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v96) ==== 56+0+5329 (secure 0 0 0) 0x7f36e409c9c0 con 0x7f36f4101320 2024-04-25T18:45:57.955 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 96 2024-04-25T18:45:57.955 INFO:teuthology.orchestra.run.smithi063.stdout:fsid e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T18:45:57.955 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-04-25T18:32:41.220967+0000 2024-04-25T18:45:57.955 INFO:teuthology.orchestra.run.smithi063.stdout:modified 2024-04-25T18:43:55.344181+0000 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:crush_version 26 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:full_ratio 0.95 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:backfillfull_ratio 0.9 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:nearfull_ratio 0.85 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:require_min_compat_client luminous 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:min_compat_client luminous 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:require_osd_release squid 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:stretch_mode_enabled false 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 92 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 92 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:max_osd 12 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6802/237733214,v1:172.21.15.63:6803/237733214] [v2:172.21.15.63:6804/237733214,v1:172.21.15.63:6805/237733214] exists,up 0615248c-d2e1-48aa-ab67-0733aaa339cc 2024-04-25T18:45:57.956 INFO:teuthology.orchestra.run.smithi063.stdout:osd.1 up in weight 1 up_from 13 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6800/135943968,v1:172.21.15.70:6801/135943968] [v2:172.21.15.70:6802/135943968,v1:172.21.15.70:6803/135943968] exists,up 2c5dd2fb-256f-461a-aa35-959c06ed8333 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6800/3090454830,v1:172.21.15.114:6801/3090454830] [v2:172.21.15.114:6802/3090454830,v1:172.21.15.114:6803/3090454830] exists,up 719d6679-f15c-4e02-b8c9-e19a26526679 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.3 up in weight 1 up_from 27 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6810/3120222680,v1:172.21.15.63:6811/3120222680] [v2:172.21.15.63:6812/3120222680,v1:172.21.15.63:6813/3120222680] exists,up 5052332b-1c7e-42e7-9ae6-4751b86b2986 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.4 up in weight 1 up_from 35 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6808/2157462947,v1:172.21.15.70:6809/2157462947] [v2:172.21.15.70:6810/2157462947,v1:172.21.15.70:6811/2157462947] exists,up 0da5a992-e540-4065-a95a-2fe0eb881da5 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.5 up in weight 1 up_from 41 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6808/4152604571,v1:172.21.15.114:6809/4152604571] [v2:172.21.15.114:6810/4152604571,v1:172.21.15.114:6811/4152604571] exists,up 8832dbef-9e3e-4abf-8ac4-d279ca3af434 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.6 up in weight 1 up_from 49 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6818/3710200907,v1:172.21.15.63:6819/3710200907] [v2:172.21.15.63:6820/3710200907,v1:172.21.15.63:6821/3710200907] exists,up ca98c42c-8187-4097-a802-137756444990 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.7 up in weight 1 up_from 54 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6816/568332364,v1:172.21.15.70:6817/568332364] [v2:172.21.15.70:6818/568332364,v1:172.21.15.70:6819/568332364] exists,up 164885fd-3ba5-4de5-9224-0de701470a27 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.8 up in weight 1 up_from 63 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6816/1230878989,v1:172.21.15.114:6817/1230878989] [v2:172.21.15.114:6818/1230878989,v1:172.21.15.114:6819/1230878989] exists,up 820c8864-3156-4c9f-b583-0360fa2455eb 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.9 up in weight 1 up_from 71 up_thru 95 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6826/2032604816,v1:172.21.15.63:6827/2032604816] [v2:172.21.15.63:6828/2032604816,v1:172.21.15.63:6829/2032604816] exists,up 5f01be5b-f17a-4fc6-a8cb-10630b362163 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.10 up in weight 1 up_from 76 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6824/3803988922,v1:172.21.15.70:6825/3803988922] [v2:172.21.15.70:6826/3803988922,v1:172.21.15.70:6827/3803988922] exists,up 067a5560-1dc0-45a8-8a1e-82061039239e 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:osd.11 up in weight 1 up_from 84 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.114:6824/3297229799,v1:172.21.15.114:6825/3297229799] [v2:172.21.15.114:6826/3297229799,v1:172.21.15.114:6827/3297229799] exists,up 10cb8d84-790d-42b3-8f29-3771c44434e4 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 2.b [2,6] 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 2.10 [2,4,8,11] 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 2.16 [2,4,7,11] 2024-04-25T18:45:57.957 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 2.1f [8,1] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.0 [5,9] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.14 [4,7] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.1a [5,6] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.1d [4,8] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.29 [3,9] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:pg_upmap_items 3.2c [4,9] 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6800/3403991527 expires 2024-04-26T18:33:52.960272+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/3648036576 expires 2024-04-26T18:33:52.960272+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1446926443 expires 2024-04-26T18:33:52.960272+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6801/3403991527 expires 2024-04-26T18:33:52.960272+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6801/2210114679 expires 2024-04-26T18:33:16.955897+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:6800/2210114679 expires 2024-04-26T18:33:16.955897+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/4120416918 expires 2024-04-26T18:33:16.955897+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1747359887 expires 2024-04-26T18:33:16.955897+0000 2024-04-25T18:45:57.958 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/1119313138 expires 2024-04-26T18:33:16.955897+0000 2024-04-25T18:45:57.959 INFO:teuthology.orchestra.run.smithi063.stdout:blocklist 172.21.15.63:0/434371372 expires 2024-04-26T18:33:52.960272+0000 2024-04-25T18:45:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.957+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 msgr2=0x7f36d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.957+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 0x7f36d80a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009810 tx=0x7f36dc019040 comp rx=0 tx=0).stop 2024-04-25T18:45:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 msgr2=0x7f36f419aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:45:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f36e400ec40 tx=0x7f36e400ef50 comp rx=0 tx=0).stop 2024-04-25T18:45:57.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 shutdown_connections 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f36d80a3c40 0x7f36d80a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f36f4101320 0x7f36f419aed0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f36f4102640 0x7f36f41a2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 --2- 172.21.15.63:0/3404760708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36f4101c80 0x7f36f419b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 >> 172.21.15.63:0/3404760708 conn(0x7f36f40fcf40 msgr2=0x7f36f41002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 shutdown_connections 2024-04-25T18:45:57.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:45:57.958+0000 7f36fa000700 1 -- 172.21.15.63:0/3404760708 wait complete. 2024-04-25T18:45:58.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:57 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:58.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:57 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3404760708' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T18:45:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:57 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:57 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3404760708' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T18:45:58.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:57 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:45:58.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:57 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3404760708' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T18:45:58.959 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T18:45:58.971 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:45:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:58 smithi114 ceph-mon[32988]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:58 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:45:59.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:58 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:45:59.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:58 smithi063 ceph-mon[28006]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:59.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:58 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:45:59.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:58 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:45:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:45:59.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:58 smithi070 ceph-mon[32820]: pgmap v448: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:45:59.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:58 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:45:59.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:58 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:46:00.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:00.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:00.293 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:45:59 smithi063 ceph-mon[28006]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:46:00.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:00.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:00.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:45:59 smithi114 ceph-mon[32988]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:46:00.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:45:59 smithi070 ceph-mon[32820]: from='mgr.14152 172.21.15.63:0/2266872602' entity='mgr.x' 2024-04-25T18:46:01.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:00 smithi114 ceph-mon[32988]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:01.335 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:00 smithi063 ceph-mon[28006]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:01.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:00 smithi070 ceph-mon[32820]: pgmap v449: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:01.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 -- 172.21.15.63:0/3809969646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109cc0 msgr2=0x7f400010dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 --2- 172.21.15.63:0/3809969646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109cc0 0x7f400010dcc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009920 tx=0x7f3fec009c30 comp rx=0 tx=0).stop 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 -- 172.21.15.63:0/3809969646 shutdown_connections 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 --2- 172.21.15.63:0/3809969646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f400010e3a0 0x7f4000115f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 --2- 172.21.15.63:0/3809969646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 --2- 172.21.15.63:0/3809969646 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4000109cc0 0x7f400010dcc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 -- 172.21.15.63:0/3809969646 >> 172.21.15.63:0/3809969646 conn(0x7f4000076210 msgr2=0x7f4000078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 -- 172.21.15.63:0/3809969646 shutdown_connections 2024-04-25T18:46:01.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.655+0000 7f4008400700 1 -- 172.21.15.63:0/3809969646 wait complete. 2024-04-25T18:46:01.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.656+0000 7f4008400700 1 Processor -- start 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 -- start start 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 0x7f40001a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 0x7f400019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4000119260 con 0x7f4000109cc0 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40001190e0 con 0x7f4000109310 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4008400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f40001193e0 con 0x7f400010e3a0 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4005000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 0x7f40001a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4006400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 0x7f400019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4005000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 0x7f40001a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48398/0 (socket says 172.21.15.63:48398) 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4005000700 1 -- 172.21.15.63:0/731544985 learned_addr learned my addr 172.21.15.63:0/731544985 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:01.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.657+0000 7f4005a00700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:01.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 -- 172.21.15.63:0/731544985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 msgr2=0x7f40001a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:01.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 -- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 msgr2=0x7f40001a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 0x7f40001a5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 -- 172.21.15.63:0/731544985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400019ff90 con 0x7f400010e3a0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4005a00700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f4006400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 0x7f400019f6b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800efc0 tx=0x7f3ff800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.658+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80092e0 con 0x7f400010e3a0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.659+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff8004bb0 con 0x7f400010e3a0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.659+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8004020 con 0x7f400010e3a0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.659+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40001a0220 con 0x7f400010e3a0 2024-04-25T18:46:01.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.659+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40001ac560 con 0x7f400010e3a0 2024-04-25T18:46:01.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.660+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40001a0590 con 0x7f400010e3a0 2024-04-25T18:46:01.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.662+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3ff801b030 con 0x7f400010e3a0 2024-04-25T18:46:01.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.663+0000 7f3ffe000700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 0x7f3fe80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:01.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.664+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f3ff80d00b0 con 0x7f400010e3a0 2024-04-25T18:46:01.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.664+0000 7f4005a00700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 0x7f3fe80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:01.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.664+0000 7f4005a00700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 0x7f3fe80a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009750 tx=0x7f3ff4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:01.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.666+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f40001a0590 con 0x7f400010e3a0 2024-04-25T18:46:01.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.945+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40001a0be0 con 0x7f400010e3a0 2024-04-25T18:46:01.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.946+0000 7f3ffe000700 1 -- 172.21.15.63:0/731544985 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7f3ff8022020 con 0x7f400010e3a0 2024-04-25T18:46:01.949 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:01.949 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:46:01.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 msgr2=0x7f3fe80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:01.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 0x7f3fe80a6180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009750 tx=0x7f3ff4009310 comp rx=0 tx=0).stop 2024-04-25T18:46:01.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 msgr2=0x7f400019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:01.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 0x7f400019f6b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800efc0 tx=0x7f3ff800c3a0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 shutdown_connections 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f3fe80a3cc0 0x7f3fe80a6180 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f400010e3a0 0x7f400019f6b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4000109310 0x7f40001a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 --2- 172.21.15.63:0/731544985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4000109cc0 0x7f40001a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.950+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 >> 172.21.15.63:0/731544985 conn(0x7f4000076210 msgr2=0x7f4000077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.951+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 shutdown_connections 2024-04-25T18:46:01.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:01.951+0000 7f4008400700 1 -- 172.21.15.63:0/731544985 wait complete. 2024-04-25T18:46:01.955 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:46:02.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:02 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/731544985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:02.343 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:02 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/731544985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:02.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:02 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/731544985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:02.823 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:46:03.304 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:03.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:03 smithi114 ceph-mon[32988]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:03.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:03 smithi063 ceph-mon[28006]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:03.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:03 smithi070 ceph-mon[32820]: pgmap v450: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:05.110 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:05 smithi063 ceph-mon[28006]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:05.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:05 smithi114 ceph-mon[32988]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:05.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:05 smithi070 ceph-mon[32820]: pgmap v451: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 -- 172.21.15.63:0/3208593791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4075590 msgr2=0x7f5fe410ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 --2- 172.21.15.63:0/3208593791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4075590 0x7f5fe410ef30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009990 tx=0x7f5fd0009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 -- 172.21.15.63:0/3208593791 shutdown_connections 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 --2- 172.21.15.63:0/3208593791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4074c70 0x7f5fe4075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 --2- 172.21.15.63:0/3208593791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe410f470 0x7f5fe4111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 --2- 172.21.15.63:0/3208593791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe4075590 0x7f5fe410ef30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 -- 172.21.15.63:0/3208593791 >> 172.21.15.63:0/3208593791 conn(0x7f5fe4079bf0 msgr2=0x7f5fe407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 -- 172.21.15.63:0/3208593791 shutdown_connections 2024-04-25T18:46:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.655+0000 7f5febe00700 1 -- 172.21.15.63:0/3208593791 wait complete. 2024-04-25T18:46:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.656+0000 7f5febe00700 1 Processor -- start 2024-04-25T18:46:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.656+0000 7f5febe00700 1 -- start start 2024-04-25T18:46:05.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.656+0000 7f5febe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe4074c70 0x7f5fe419b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:05.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.656+0000 7f5febe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 0x7f5fe419b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:05.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.656+0000 7f5febe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5febe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe4114ec0 con 0x7f5fe4074c70 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5febe00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe4114d40 con 0x7f5fe410f470 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5febe00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe4115040 con 0x7f5fe4075590 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe4074c70 0x7f5fe419b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 0x7f5fe419b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59654/0 (socket says 172.21.15.63:59654) 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe9e00700 1 -- 172.21.15.63:0/4144384798 learned_addr learned my addr 172.21.15.63:0/4144384798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 -- 172.21.15.63:0/4144384798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 msgr2=0x7f5fe419fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 -- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe4074c70 msgr2=0x7f5fe419b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe4074c70 0x7f5fe419b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe8a00700 1 -- 172.21.15.63:0/4144384798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe41a04c0 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.657+0000 7f5fe9e00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5fe8a00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 0x7f5fe419b9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009990 tx=0x7f5fd00057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0027070 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd0022470 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe41a0750 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0017650 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.658+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe41a8030 con 0x7f5fe4075590 2024-04-25T18:46:05.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.659+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe4069730 con 0x7f5fe4075590 2024-04-25T18:46:05.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.661+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5fd002a070 con 0x7f5fe4075590 2024-04-25T18:46:05.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.663+0000 7f5fe2000700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 0x7f5fcc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:05.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.663+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f5fd00d0ff0 con 0x7f5fe4075590 2024-04-25T18:46:05.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.663+0000 7f5fe9400700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 0x7f5fcc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:05.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.664+0000 7f5fe9400700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 0x7f5fcc0a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009c60 tx=0x7f5fd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:05.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.665+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5fd0013070 con 0x7f5fe4075590 2024-04-25T18:46:05.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.948+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5fe402df80 con 0x7f5fe4075590 2024-04-25T18:46:05.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.950+0000 7f5fe2000700 1 -- 172.21.15.63:0/4144384798 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11868 (secure 0 0 0) 0x7f5fd009d9f0 con 0x7f5fe4075590 2024-04-25T18:46:05.952 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:05.952 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:42:57.440972+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:46:05.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 msgr2=0x7f5fcc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 0x7f5fcc0a6180 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8009c60 tx=0x7f5fd8009260 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 msgr2=0x7f5fe419b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 0x7f5fe419b9f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009990 tx=0x7f5fd00057c0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 shutdown_connections 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5fcc0a3cc0 0x7f5fcc0a6180 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5fe410f470 0x7f5fe419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fe4074c70 0x7f5fe419b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 --2- 172.21.15.63:0/4144384798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5fe4075590 0x7f5fe419b9f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 >> 172.21.15.63:0/4144384798 conn(0x7f5fe4079bf0 msgr2=0x7f5fe4101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 shutdown_connections 2024-04-25T18:46:05.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:05.953+0000 7f5febe00700 1 -- 172.21.15.63:0/4144384798 wait complete. 2024-04-25T18:46:05.957 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 23 2024-04-25T18:46:06.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:06 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4144384798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:06.334 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:06 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4144384798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:06 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4144384798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:06.796 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:46:07.288 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:07.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:07 smithi114 ceph-mon[32988]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:07.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:07 smithi063 ceph-mon[28006]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:07.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:07 smithi070 ceph-mon[32820]: pgmap v452: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:09.221 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:09 smithi063 ceph-mon[28006]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:09.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:09 smithi114 ceph-mon[32988]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:09.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:09 smithi070 ceph-mon[32820]: pgmap v453: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:09.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 -- 172.21.15.63:0/2633802983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee010f4e0 msgr2=0x7faee01118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:09.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 --2- 172.21.15.63:0/2633802983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee010f4e0 0x7faee01118d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faed8009f90 tx=0x7faed800b330 comp rx=0 tx=0).stop 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 -- 172.21.15.63:0/2633802983 shutdown_connections 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 --2- 172.21.15.63:0/2633802983 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee010f4e0 0x7faee01118d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 --2- 172.21.15.63:0/2633802983 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee006afe0 0x7faee006b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 --2- 172.21.15.63:0/2633802983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006b900 0x7faee010efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 -- 172.21.15.63:0/2633802983 >> 172.21.15.63:0/2633802983 conn(0x7faee00788b0 msgr2=0x7faee0078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 -- 172.21.15.63:0/2633802983 shutdown_connections 2024-04-25T18:46:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.713+0000 7faee5000700 1 -- 172.21.15.63:0/2633802983 wait complete. 2024-04-25T18:46:09.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.714+0000 7faee5000700 1 Processor -- start 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 -- start start 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 0x7faee019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee010f4e0 0x7faee019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faee0114e30 con 0x7faee006afe0 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faee0114cb0 con 0x7faee010f4e0 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faee5000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faee0114fb0 con 0x7faee006b900 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faedde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 0x7faee019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:09.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faedd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:09.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faedde00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 0x7faee019b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46692/0 (socket says 172.21.15.63:46692) 2024-04-25T18:46:09.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faedde00700 1 -- 172.21.15.63:0/2746024645 learned_addr learned my addr 172.21.15.63:0/2746024645 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:09.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.715+0000 7faedd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59694/0 (socket says 172.21.15.63:59694) 2024-04-25T18:46:09.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faede800700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee010f4e0 0x7faee019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 -- 172.21.15.63:0/2746024645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee010f4e0 msgr2=0x7faee019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee010f4e0 0x7faee019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 -- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 msgr2=0x7faee019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 0x7faee019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 -- 172.21.15.63:0/2746024645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee01a0310 con 0x7faee006b900 2024-04-25T18:46:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.716+0000 7faedd400700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faecc00eb90 tx=0x7faecc00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.717+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc00ccf0 con 0x7faee006b900 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.717+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee01a0600 con 0x7faee006b900 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.717+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faecc0044d0 con 0x7faee006b900 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.717+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc010430 con 0x7faee006b900 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.717+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee01a7e80 con 0x7faee006b900 2024-04-25T18:46:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.718+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee0069730 con 0x7faee006b900 2024-04-25T18:46:09.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.720+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7faecc003680 con 0x7faee006b900 2024-04-25T18:46:09.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.722+0000 7faed6a00700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 0x7faec40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:09.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.722+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7faecc0d07c0 con 0x7faee006b900 2024-04-25T18:46:09.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.722+0000 7faedde00700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 0x7faec40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:09.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.723+0000 7faedde00700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 0x7faec40a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faed0009d20 tx=0x7faed0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:09.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.724+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faecc09d110 con 0x7faee006b900 2024-04-25T18:46:09.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.958+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faee002df80 con 0x7faee006b900 2024-04-25T18:46:09.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.961+0000 7faed6a00700 1 -- 172.21.15.63:0/2746024645 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+22736 (secure 0 0 0) 0x7faecc09ca10 con 0x7faee006b900 2024-04-25T18:46:09.962 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:09.963 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":96,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:43:55.344181+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:46:09.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 msgr2=0x7faec40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 0x7faec40a6100 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faed0009d20 tx=0x7faed0009260 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 msgr2=0x7faee019b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faecc00eb90 tx=0x7faecc00eea0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 shutdown_connections 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7faec40a3c40 0x7faec40a6100 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faee010f4e0 0x7faee019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faee006afe0 0x7faee019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 --2- 172.21.15.63:0/2746024645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faee006b900 0x7faee019b840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 >> 172.21.15.63:0/2746024645 conn(0x7faee00788b0 msgr2=0x7faee0110950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 shutdown_connections 2024-04-25T18:46:09.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:09.963+0000 7faee5000700 1 -- 172.21.15.63:0/2746024645 wait complete. 2024-04-25T18:46:10.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:10 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2746024645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:46:10.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:10 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2746024645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:46:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:10 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2746024645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:46:10.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T18:42:31.925526+0000', 'modified': '2024-04-25T18:42:57.440972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24533, 'mds_1': 24520, 'mds_2': 24508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/3182056159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 3182056159}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 3182056159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6837/4043589496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 4043589496}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 4043589496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/3172988525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3172988525}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 3172988525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.114:6837/1526734616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 1526734616}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1526734616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.114:6839/2417088150', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 2417088150}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 2417088150}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.70:6839/1944035005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 1944035005}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 1944035005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24533, 24508, 24520]} 2024-04-25T18:46:10.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T18:46:10.868 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-25T18:46:11.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:11 smithi063 ceph-mon[28006]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:11.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:11 smithi114 ceph-mon[32988]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:11.333 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:11.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:11 smithi070 ceph-mon[32820]: pgmap v454: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:12.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:12 smithi063 ceph-mon[28006]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:12.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:12 smithi114 ceph-mon[32988]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:12 smithi070 ceph-mon[32820]: pgmap v455: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:13.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.756+0000 7f42d6000700 1 -- 172.21.15.63:0/1906531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d006b4f0 msgr2=0x7f42d006b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.756+0000 7f42d6000700 1 --2- 172.21.15.63:0/1906531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d006b4f0 0x7f42d006b970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009990 tx=0x7f42bc009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.756+0000 7f42d6000700 1 -- 172.21.15.63:0/1906531706 shutdown_connections 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 --2- 172.21.15.63:0/1906531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d01053c0 0x7f42d0105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 --2- 172.21.15.63:0/1906531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 0x7f42d006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 --2- 172.21.15.63:0/1906531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d006b4f0 0x7f42d006b970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 -- 172.21.15.63:0/1906531706 >> 172.21.15.63:0/1906531706 conn(0x7f42d0078780 msgr2=0x7f42d0078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 -- 172.21.15.63:0/1906531706 shutdown_connections 2024-04-25T18:46:13.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.757+0000 7f42d6000700 1 -- 172.21.15.63:0/1906531706 wait complete. 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.758+0000 7f42d6000700 1 Processor -- start 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.758+0000 7f42d6000700 1 -- start start 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.758+0000 7f42d6000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d006b4f0 0x7f42d019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d6000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d6000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 0x7f42d01a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d6000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0119060 con 0x7f42d01053c0 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d6000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0118ee0 con 0x7f42d0107920 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d6000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f42d01191e0 con 0x7f42d006b4f0 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46714/0 (socket says 172.21.15.63:46714) 2024-04-25T18:46:13.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 -- 172.21.15.63:0/676370062 learned_addr learned my addr 172.21.15.63:0/676370062 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:13.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d4c00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d006b4f0 0x7f42d019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42d5600700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 0x7f42d01a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 -- 172.21.15.63:0/676370062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d006b4f0 msgr2=0x7f42d019f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d006b4f0 0x7f42d019f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 -- 172.21.15.63:0/676370062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 msgr2=0x7f42d01a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 0x7f42d01a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.759+0000 7f42cfe00700 1 -- 172.21.15.63:0/676370062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d01a44f0 con 0x7f42d01053c0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42cfe00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f42bc000c00 tx=0x7f42bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42cd600700 1 -- 172.21.15.63:0/676370062 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc027070 con 0x7f42d01053c0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42cd600700 1 -- 172.21.15.63:0/676370062 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42bc003f20 con 0x7f42d01053c0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42cd600700 1 -- 172.21.15.63:0/676370062 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc022ca0 con 0x7f42d01053c0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d01a4780 con 0x7f42d01053c0 2024-04-25T18:46:13.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.760+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d01ac060 con 0x7f42d01053c0 2024-04-25T18:46:13.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.764+0000 7f42cd600700 1 -- 172.21.15.63:0/676370062 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f42bc02a070 con 0x7f42d01053c0 2024-04-25T18:46:13.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.765+0000 7f42cd600700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 0x7f42b80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.765+0000 7f42e1f5ab80 1 Processor -- start 2024-04-25T18:46:13.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.765+0000 7f42cd600700 1 -- 172.21.15.63:0/676370062 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f42bc0d1200 con 0x7f42d01053c0 2024-04-25T18:46:13.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.765+0000 7f42d4c00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 0x7f42b80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.765+0000 7f42e1f5ab80 1 -- start start 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115fa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 0x559e115fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e11542e40 con 0x559e115ef3e0 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559e114e96e0 con 0x559e115efd00 2024-04-25T18:46:13.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e1153ee00 con 0x559e115fac40 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42d4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42d4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46728/0 (socket says 172.21.15.63:46728) 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42d4c00700 1 -- 172.21.15.63:0/1788596366 learned_addr learned my addr 172.21.15.63:0/1788596366 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42cfe00700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115fa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42d5600700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 0x559e115fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.766+0000 7f42d4c00700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 0x7f42b80a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005950 tx=0x7f42c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 -- 172.21.15.63:0/1788596366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 msgr2=0x559e115fa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115fa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 -- 172.21.15.63:0/1788596366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 msgr2=0x559e115ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 -- 172.21.15.63:0/1788596366 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e114df630 con 0x559e115fac40 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d4c00700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:13.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42d5600700 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 0x559e115fd030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f42c8003f10 tx=0x7f42c8009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5478bbd9b3083843 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42c2000700 1 -- 172.21.15.63:0/1788596366 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8004bb0 con 0x559e115fac40 2024-04-25T18:46:13.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42c2000700 1 -- 172.21.15.63:0/1788596366 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c80048a0 con 0x559e115fac40 2024-04-25T18:46:13.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.767+0000 7f42c2000700 1 -- 172.21.15.63:0/1788596366 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c8007d60 con 0x559e115fac40 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/1788596366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 msgr2=0x559e115fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 0x559e115fd030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f42c8003f10 tx=0x7f42c8009ba0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/1788596366 shutdown_connections 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115fac40 0x559e115fd030 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e115ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.771 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/1788596366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115fa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/1788596366 >> 172.21.15.63:0/1788596366 conn(0x559e1151e550 msgr2=0x559e115eba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.768+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/1788596366 shutdown_connections 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.769+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/1788596366 wait complete. 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.769+0000 7f42e1f5ab80 1 -- start start 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.769+0000 7f42e1f5ab80 1 Processor -- start 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115ef3e0 0x559e115fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e115fd7c0 con 0x559e115fac40 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559e114fc810 con 0x559e115efd00 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e115fd940 con 0x559e115ef3e0 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42e1f5ab80 10 client.0 _ll_register_callbacks cb 0x559e114db6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42d5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42d5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46732/0 (socket says 172.21.15.63:46732) 2024-04-25T18:46:13.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42d4c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115ef3e0 0x559e115fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42d5600700 1 -- 172.21.15.63:0/2403159440 learned_addr learned my addr 172.21.15.63:0/2403159440 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:13.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.770+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 -- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115ef3e0 msgr2=0x559e115fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115ef3e0 0x559e115fa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 -- 172.21.15.63:0/2403159440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 msgr2=0x559e115f7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 -- 172.21.15.63:0/2403159440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e11490880 con 0x559e115efd00 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115f76b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f42bc099940 tx=0x7f42bc09da20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1aba9abb3749182 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc09dbe0 con 0x559e115efd00 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f42bc09b7e0 con 0x559e115efd00 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42c80048a0 con 0x559e115efd00 2024-04-25T18:46:13.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f42bc09d4e0 con 0x559e115efd00 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42e1f5ab80 10 client.24995 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.771+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e114d9aa0 con 0x559e115efd00 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42c80048a0 con 0x559e115efd00 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f42bc0b6bd0 con 0x559e115efd00 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 10 client.24995 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 10 client.24995 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 20 client.24995 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e1160a4d0 0x559e1160b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 4 client.24995 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e11550630 con 0x559e1160a4d0 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.772+0000 7f42d4c00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e1160a4d0 0x559e1160b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.773+0000 7f42d4c00700 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e1160a4d0 0x559e1160b940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab829d1f6a2b46b9 server_cookie=57801004d33890cc in_seq=0 out_seq=0 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.773+0000 7f42b6a00700 10 client.24995 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42b6a00700 1 -- 172.21.15.63:0/2403159440 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559e11550630 con 0x559e1160a4d0 2024-04-25T18:46:13.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42b6a00700 10 client.24995 handle_command_reply: tid=0 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 client.24995 shutdown 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 msgr2=0x559e115f76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115f76b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f42bc099940 tx=0x7f42bc09da20 comp rx=0 tx=0).stop 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 shutdown_connections 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e1160a4d0 0x559e1160b940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115ef3e0 0x559e115fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42b6a00700 0 client.24995 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2403159440 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115efd00 0x559e115f76b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.774+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 >> 172.21.15.63:0/2403159440 conn(0x559e1151e550 msgr2=0x559e115eb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.775+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 shutdown_connections 2024-04-25T18:46:13.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.775+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2403159440 wait complete. 2024-04-25T18:46:13.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.775+0000 7f42e1f5ab80 20 client.24995 trim_cache size 0 max 16384 2024-04-25T18:46:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.797+0000 7f42e1f5ab80 1 Processor -- start 2024-04-25T18:46:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 -- start start 2024-04-25T18:46:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115ef3e0 0x559e1160b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e115f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e11609c80 con 0x559e115fac40 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559e1160a350 con 0x559e115ef3e0 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e1160aa20 con 0x559e115efd00 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42d5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42d5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46738/0 (socket says 172.21.15.63:46738) 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42d5600700 1 -- 172.21.15.63:0/936356521 learned_addr learned my addr 172.21.15.63:0/936356521 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42d4c00700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115ef3e0 0x559e1160b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.798+0000 7f42cfe00700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e115f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 -- 172.21.15.63:0/936356521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115ef3e0 msgr2=0x559e1160b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115ef3e0 0x559e1160b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 -- 172.21.15.63:0/936356521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 msgr2=0x559e115f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 -- 172.21.15.63:0/936356521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e1160a170 con 0x559e115efd00 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42d5600700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42cfe00700 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e115f5fa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42bc09db40 tx=0x7f42bc00b560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa014edeeff3c42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42c1600700 1 -- 172.21.15.63:0/936356521 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc082070 con 0x559e115efd00 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42c1600700 1 -- 172.21.15.63:0/936356521 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42bc0bc070 con 0x559e115efd00 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.799+0000 7f42c1600700 1 -- 172.21.15.63:0/936356521 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc09b7e0 con 0x559e115efd00 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/936356521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 msgr2=0x559e115f5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e115f5fa0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f42bc09db40 tx=0x7f42bc00b560 comp rx=0 tx=0).stop 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/936356521 shutdown_connections 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115fac40 0x559e115f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115ef3e0 0x559e1160b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/936356521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e115f5fa0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/936356521 >> 172.21.15.63:0/936356521 conn(0x559e1151e550 msgr2=0x559e115f7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/936356521 shutdown_connections 2024-04-25T18:46:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.800+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/936356521 wait complete. 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.801+0000 7f42e1f5ab80 1 -- start start 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.801+0000 7f42e1f5ab80 1 Processor -- start 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e1160efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e1160f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559e1160b7c0 con 0x559e115ef3e0 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559e1160b0f0 con 0x559e115fac40 2024-04-25T18:46:13.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559e1160be90 con 0x559e115efd00 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42e1f5ab80 10 client.0 _ll_register_callbacks cb 0x559e114db6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d5600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d4c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e1160efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e1160f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d5600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46486/0 (socket says 172.21.15.63:46486) 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d5600700 1 -- 172.21.15.63:0/2493552542 learned_addr learned my addr 172.21.15.63:0/2493552542 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 -- 172.21.15.63:0/2493552542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 msgr2=0x559e11613620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 -- 172.21.15.63:0/2493552542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 msgr2=0x559e1160efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e1160efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42cfe00700 1 -- 172.21.15.63:0/2493552542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e11613c60 con 0x559e115efd00 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d5600700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.802+0000 7f42d4c00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e1160efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42cfe00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e1160f510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f42bc09db40 tx=0x7f42bc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=860f6ecb262a3cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42bc09dbe0 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f42bc09b7e0 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42bc0bc070 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f42bc09c930 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42e1f5ab80 10 client.25012 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x559e11671f90 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.803+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42bc0bc070 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f42bc09c3a0 con 0x559e115efd00 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 10 client.25012 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 10 client.25012 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 20 client.25012 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e116766f0 0x559e11675400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 4 client.25012 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:46:13.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "status"}) -- 0x559e1154a6d0 con 0x559e116766f0 2024-04-25T18:46:13.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.804+0000 7f42d4c00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e116766f0 0x559e11675400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42d4c00700 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e116766f0 0x559e11675400 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1e4ce45c1b4da06 server_cookie=d70e533a7ebead3 in_seq=0 out_seq=0 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42b6a00700 10 client.25012 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42b6a00700 1 -- 172.21.15.63:0/2493552542 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559e1154a6d0 con 0x559e116766f0 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42b6a00700 10 client.25012 handle_command_reply: tid=0 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 client.25012 shutdown 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 msgr2=0x559e1160f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e1160f510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f42bc09db40 tx=0x7f42bc009310 comp rx=0 tx=0).stop 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 shutdown_connections 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x559e116766f0 0x559e11675400 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559e115fac40 0x559e11613620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42b6a00700 0 client.25012 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559e115ef3e0 0x559e1160efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 --2- 172.21.15.63:0/2493552542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559e115efd00 0x559e1160f510 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 >> 172.21.15.63:0/2493552542 conn(0x559e1151e550 msgr2=0x559e115eb4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 shutdown_connections 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.805+0000 7f42e1f5ab80 1 -- 172.21.15.63:0/2493552542 wait complete. 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.806+0000 7f42e1f5ab80 20 client.25012 trim_cache size 0 max 16384 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:46:13.810 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 2, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24508, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 203.16304627100001, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 262.95358685500003 2024-04-25T18:46:13.811 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.806+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 msgr2=0x7f42b80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.806+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 0x7f42b80a6180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005950 tx=0x7f42c400b410 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.806+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 msgr2=0x7f42d019fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.806+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f42bc000c00 tx=0x7f42bc0047b0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 shutdown_connections 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f42b80a3cc0 0x7f42b80a6180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f42d006b4f0 0x7f42d019f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42d0107920 0x7f42d01a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 --2- 172.21.15.63:0/676370062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42d01053c0 0x7f42d019fa70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 >> 172.21.15.63:0/676370062 conn(0x7f42d0078780 msgr2=0x7f42d0100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:13.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 shutdown_connections 2024-04-25T18:46:13.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:13.807+0000 7f42d6000700 1 -- 172.21.15.63:0/676370062 wait complete. 2024-04-25T18:46:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:14 smithi114 ceph-mon[32988]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:14 smithi063 ceph-mon[28006]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:14 smithi070 ceph-mon[32820]: pgmap v456: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-25T18:46:14.791 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T18:46:15.252 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:16 smithi114 ceph-mon[32988]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:16 smithi063 ceph-mon[28006]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:16 smithi070 ceph-mon[32820]: pgmap v457: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-25T18:46:17.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 -- 172.21.15.63:0/287957653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 msgr2=0x7f5e30111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 --2- 172.21.15.63:0/287957653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e30111b90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800b0d0 tx=0x7f5e2800b3e0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 -- 172.21.15.63:0/287957653 shutdown_connections 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 --2- 172.21.15.63:0/287957653 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e30111b90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 --2- 172.21.15.63:0/287957653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e301052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 --2- 172.21.15.63:0/287957653 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e301098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 -- 172.21.15.63:0/287957653 >> 172.21.15.63:0/287957653 conn(0x7f5e30100770 msgr2=0x7f5e30102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 -- 172.21.15.63:0/287957653 shutdown_connections 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.596+0000 7f5e34e00700 1 -- 172.21.15.63:0/287957653 wait complete. 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.597+0000 7f5e34e00700 1 Processor -- start 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.597+0000 7f5e34e00700 1 -- start start 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e3019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e3019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5e30114e60 con 0x7f5e30104ee0 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e30114ce0 con 0x7f5e30105890 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e34e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e30114fe0 con 0x7f5e30109fc0 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42480/0 (socket says 172.21.15.63:42480) 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2de00700 1 -- 172.21.15.63:0/3473635028 learned_addr learned my addr 172.21.15.63:0/3473635028 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2d400700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e3019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2e800700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e3019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.598+0000 7f5e2de00700 1 -- 172.21.15.63:0/3473635028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 msgr2=0x7f5e3019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2de00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e3019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2de00700 1 -- 172.21.15.63:0/3473635028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 msgr2=0x7f5e3019b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2de00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e3019b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2de00700 1 -- 172.21.15.63:0/3473635028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e301a02c0 con 0x7f5e30104ee0 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2d400700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e3019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e2de00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e9a0 tx=0x7f5e2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e26a00700 1 -- 172.21.15.63:0/3473635028 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000ca10 con 0x7f5e30104ee0 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e26a00700 1 -- 172.21.15.63:0/3473635028 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e2000cbb0 con 0x7f5e30104ee0 2024-04-25T18:46:17.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e301a05b0 con 0x7f5e30104ee0 2024-04-25T18:46:17.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e26a00700 1 -- 172.21.15.63:0/3473635028 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20018680 con 0x7f5e30104ee0 2024-04-25T18:46:17.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.599+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e301a7e30 con 0x7f5e30104ee0 2024-04-25T18:46:17.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 Processor -- start 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 -- start start 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d290a0 0x5569f9d229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5569f9c74e40 con 0x5569f9d22ef0 2024-04-25T18:46:17.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5569f9c1b6e0 con 0x5569f9d290a0 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.604+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569f9c70e00 con 0x5569f9d28780 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54446/0 (socket says 172.21.15.63:54446) 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42486/0 (socket says 172.21.15.63:42486) 2024-04-25T18:46:17.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 -- 172.21.15.63:0/1921108268 learned_addr learned my addr 172.21.15.63:0/1921108268 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e26a00700 1 -- 172.21.15.63:0/3473635028 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5e20010430 con 0x7f5e30104ee0 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 -- 172.21.15.63:0/1921108268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d290a0 msgr2=0x5569f9d229b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d290a0 0x5569f9d229b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 -- 172.21.15.63:0/1921108268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 msgr2=0x5569f9d23330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2e800700 1 -- 172.21.15.63:0/1921108268 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f9c11630 con 0x5569f9d28780 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.605+0000 7f5e2d400700 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.606+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e280097d0 tx=0x7f5e28009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561fec023b00df2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.606+0000 7f5e1b400700 1 -- 172.21.15.63:0/1921108268 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2800baa0 con 0x5569f9d28780 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.606+0000 7f5e1b400700 1 -- 172.21.15.63:0/1921108268 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e28020030 con 0x5569f9d28780 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.606+0000 7f5e1b400700 1 -- 172.21.15.63:0/1921108268 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28005450 con 0x5569f9d28780 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e26a00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 0x7f5e140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1921108268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 msgr2=0x5569f9d28b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5e280097d0 tx=0x7f5e28009800 comp rx=0 tx=0).stop 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e26a00700 1 -- 172.21.15.63:0/3473635028 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f5e200d06d0 con 0x7f5e30104ee0 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1921108268 shutdown_connections 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d28b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d22ef0 0x5569f9d23330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1921108268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d290a0 0x5569f9d229b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1921108268 >> 172.21.15.63:0/1921108268 conn(0x5569f9c50f00 msgr2=0x5569f9d1d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.607+0000 7f5e2de00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 0x7f5e140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.608+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1921108268 shutdown_connections 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.608+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1921108268 wait complete. 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.608+0000 7f5e2de00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 0x7f5e140a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5e20098de0 tx=0x7f5e20009790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.608+0000 7f5e3f6fab80 1 -- start start 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 Processor -- start 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d22ef0 0x5569f9d2eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d2c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d2f8c0 con 0x5569f9d2c4e0 2024-04-25T18:46:17.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5569f9c2e810 con 0x5569f9d22ef0 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d2fa40 con 0x5569f9d28780 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e3f6fab80 10 client.0 _ll_register_callbacks cb 0x5569f9c0d6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54452/0 (socket says 172.21.15.63:54452) 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.609+0000 7f5e2d400700 1 -- 172.21.15.63:0/3225571329 learned_addr learned my addr 172.21.15.63:0/3225571329 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:17.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2e800700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d2c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2e800700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d22ef0 0x5569f9d2eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 -- 172.21.15.63:0/3225571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d22ef0 msgr2=0x5569f9d2eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d22ef0 0x5569f9d2eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 -- 172.21.15.63:0/3225571329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 msgr2=0x5569f9d2c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d2c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 -- 172.21.15.63:0/3225571329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f9bc2880 con 0x5569f9d28780 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e2d400700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c012d20 tx=0x7f5e1c00f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5ac11c3710dc4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e0fe00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0120d0 con 0x5569f9d28780 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f5e1c012e00 con 0x5569f9d28780 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.610+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e28020030 con 0x5569f9d28780 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.611+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5e1c012270 con 0x5569f9d28780 2024-04-25T18:46:17.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.611+0000 7f5e3f6fab80 10 client.25024 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.611+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5569f9c0baa0 con 0x5569f9d28780 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.611+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e28020030 con 0x5569f9d28780 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.611+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5e1c010550 con 0x5569f9d28780 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 10 client.25024 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 10 client.25024 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 20 client.25024 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9d3ca40 0x5569f9d3deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 4 client.25024 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569f9c82630 con 0x5569f9d3ca40 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.612+0000 7f5e2e800700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9d3ca40 0x5569f9d3deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.613+0000 7f5e2e800700 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9d3ca40 0x5569f9d3deb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbed1cbde842fb00 server_cookie=ffc9e890c263456d in_seq=0 out_seq=0 2024-04-25T18:46:17.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.613+0000 7f5e0fe00700 10 client.25024 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.613+0000 7f5e0fe00700 1 -- 172.21.15.63:0/3225571329 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5569f9c82630 con 0x5569f9d3ca40 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.613+0000 7f5e0fe00700 10 client.25024 handle_command_reply: tid=0 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.613+0000 7f5e3f6fab80 1 client.25024 shutdown 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 msgr2=0x5569f9d2bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c012d20 tx=0x7f5e1c00f6a0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 shutdown_connections 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9d3ca40 0x5569f9d3deb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d2c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d22ef0 0x5569f9d2eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/3225571329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d28780 0x5569f9d2bfa0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e0fe00700 0 client.25024 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 >> 172.21.15.63:0/3225571329 conn(0x5569f9c50f00 msgr2=0x5569f9d1ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 shutdown_connections 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/3225571329 wait complete. 2024-04-25T18:46:17.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.614+0000 7f5e3f6fab80 20 client.25024 trim_cache size 0 max 16384 2024-04-25T18:46:17.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.636+0000 7f5e3f6fab80 1 Processor -- start 2024-04-25T18:46:17.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.636+0000 7f5e3f6fab80 1 -- start start 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d1d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 0x5569f9d47310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3c1f0 con 0x5569f9d2c4e0 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3c8c0 con 0x5569f9d28780 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3cf90 con 0x5569f9d22ef0 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 0x5569f9d47310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42494/0 (socket says 172.21.15.63:42494) 2024-04-25T18:46:17.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 -- 172.21.15.63:0/1719910818 learned_addr learned my addr 172.21.15.63:0/1719910818 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:17.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d1d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 -- 172.21.15.63:0/1719910818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 msgr2=0x5569f9d1d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d1d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 -- 172.21.15.63:0/1719910818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 msgr2=0x5569f9d47310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 0x5569f9d47310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2e800700 1 -- 172.21.15.63:0/1719910818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f9d3ee10 con 0x5569f9d2c4e0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.637+0000 7f5e2d400700 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 0x5569f9d47310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.638+0000 7f5e2e800700 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e28005bd0 tx=0x7f5e2801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd478b085300260 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.638+0000 7f5e1aa00700 1 -- 172.21.15.63:0/1719910818 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2802f040 con 0x5569f9d2c4e0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.638+0000 7f5e1aa00700 1 -- 172.21.15.63:0/1719910818 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e28020040 con 0x5569f9d2c4e0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.638+0000 7f5e1aa00700 1 -- 172.21.15.63:0/1719910818 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28005410 con 0x5569f9d2c4e0 2024-04-25T18:46:17.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1719910818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 msgr2=0x5569f9d47850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e28005bd0 tx=0x7f5e2801e040 comp rx=0 tx=0).stop 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1719910818 shutdown_connections 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d1d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d2c4e0 0x5569f9d47850 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/1719910818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d28780 0x5569f9d47310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1719910818 >> 172.21.15.63:0/1719910818 conn(0x5569f9c50f00 msgr2=0x5569f9d260e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1719910818 shutdown_connections 2024-04-25T18:46:17.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/1719910818 wait complete. 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.639+0000 7f5e3f6fab80 1 -- start start 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 Processor -- start 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d28780 0x5569f9d422e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d2c4e0 0x5569f9d42820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3dd30 con 0x5569f9d28780 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3d660 con 0x5569f9d2c4e0 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5569f9d3e400 con 0x5569f9d22ef0 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e3f6fab80 10 client.0 _ll_register_callbacks cb 0x5569f9c0d6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:17.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e2d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d28780 0x5569f9d422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e2e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54466/0 (socket says 172.21.15.63:54466) 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.640+0000 7f5e2e800700 1 -- 172.21.15.63:0/436236914 learned_addr learned my addr 172.21.15.63:0/436236914 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d2c4e0 0x5569f9d42820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 -- 172.21.15.63:0/436236914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d2c4e0 msgr2=0x5569f9d42820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d2c4e0 0x5569f9d42820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 -- 172.21.15.63:0/436236914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d28780 msgr2=0x5569f9d422e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d28780 0x5569f9d422e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 -- 172.21.15.63:0/436236914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f9d47b30 con 0x5569f9d22ef0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e2e800700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800cff0 tx=0x7f5e280099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7d570855c74f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e0fe00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e28003a00 con 0x5569f9d22ef0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f5e28003ba0 con 0x5569f9d22ef0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e28020040 con 0x5569f9d22ef0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.641+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f5e28005190 con 0x5569f9d22ef0 2024-04-25T18:46:17.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 10 client.25030 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5569f9da4690 con 0x5569f9d22ef0 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e28020040 con 0x5569f9d22ef0 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f5e2803b070 con 0x5569f9d22ef0 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 10 client.25030 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 10 client.25030 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 20 client.25030 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9da8b80 0x5569f9da7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 4 client.25030 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:46:17.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "status"}) -- 0x5569f9c7c6d0 con 0x5569f9da8b80 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.642+0000 7f5e2d400700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9da8b80 0x5569f9da7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e2d400700 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9da8b80 0x5569f9da7890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c60d9e6638015720 server_cookie=2f4740172064ff71 in_seq=0 out_seq=0 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e0fe00700 10 client.25030 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e0fe00700 1 -- 172.21.15.63:0/436236914 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5569f9c7c6d0 con 0x5569f9da8b80 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e0fe00700 10 client.25030 handle_command_reply: tid=0 2024-04-25T18:46:17.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 client.25030 shutdown 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 msgr2=0x5569f9d47330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800cff0 tx=0x7f5e280099e0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 shutdown_connections 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5569f9da8b80 0x5569f9da7890 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5569f9d22ef0 0x5569f9d47330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5569f9d2c4e0 0x5569f9d42820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 --2- 172.21.15.63:0/436236914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569f9d28780 0x5569f9d422e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e0fe00700 0 client.25030 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 >> 172.21.15.63:0/436236914 conn(0x5569f9c50f00 msgr2=0x5569f9d29490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.643+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 shutdown_connections 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e3f6fab80 1 -- 172.21.15.63:0/436236914 wait complete. 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e3f6fab80 20 client.25030 trim_cache size 0 max 16384 2024-04-25T18:46:17.647 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 1, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24520, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 208.01257009099999, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 261.49776973799999 2024-04-25T18:46:17.648 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 msgr2=0x7f5e140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 0x7f5e140a6100 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5e20098de0 tx=0x7f5e20009790 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 msgr2=0x7f5e3019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.644+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e9a0 tx=0x7f5e2000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 shutdown_connections 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f5e140a3c40 0x7f5e140a6100 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e30109fc0 0x7f5e3019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e30104ee0 0x7f5e3019b2b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 --2- 172.21.15.63:0/3473635028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e30105890 0x7f5e3019b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:17.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 >> 172.21.15.63:0/3473635028 conn(0x7f5e30100770 msgr2=0x7f5e30102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:17.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 shutdown_connections 2024-04-25T18:46:17.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:17.645+0000 7f5e34e00700 1 -- 172.21.15.63:0/3473635028 wait complete. 2024-04-25T18:46:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:18 smithi063 ceph-mon[28006]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:18 smithi114 ceph-mon[32988]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:18.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T18:46:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:18 smithi070 ceph-mon[32820]: pgmap v458: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:19.067 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:20 smithi063 ceph-mon[28006]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:20.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:20 smithi114 ceph-mon[32988]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:20 smithi070 ceph-mon[32820]: pgmap v459: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 -- 172.21.15.63:0/3239646989 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 msgr2=0x7f0618109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 --2- 172.21.15.63:0/3239646989 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f0618109400 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f06040098f0 tx=0x7f0604009c00 comp rx=0 tx=0).stop 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 -- 172.21.15.63:0/3239646989 shutdown_connections 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 --2- 172.21.15.63:0/3239646989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f0618115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 --2- 172.21.15.63:0/3239646989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 0x7f061810f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 --2- 172.21.15.63:0/3239646989 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f0618109400 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 -- 172.21.15.63:0/3239646989 >> 172.21.15.63:0/3239646989 conn(0x7f06180787d0 msgr2=0x7f0618107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 -- 172.21.15.63:0/3239646989 shutdown_connections 2024-04-25T18:46:21.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.475+0000 7f061f200700 1 -- 172.21.15.63:0/3239646989 wait complete. 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.476+0000 7f061f200700 1 Processor -- start 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 -- start start 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f061819fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 0x7f06181a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0618119120 con 0x7f0618108f80 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0618118fa0 con 0x7f061810f030 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061f200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06181192a0 con 0x7f0618109940 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.477+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42520/0 (socket says 172.21.15.63:42520) 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061de00700 1 -- 172.21.15.63:0/1834488773 learned_addr learned my addr 172.21.15.63:0/1834488773 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f061819fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061e800700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 0x7f06181a3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 -- 172.21.15.63:0/1834488773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 msgr2=0x7f06181a3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 0x7f06181a3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 -- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 msgr2=0x7f061819f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 -- 172.21.15.63:0/1834488773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06181a4580 con 0x7f0618109940 2024-04-25T18:46:21.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061de00700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.478+0000 7f061d400700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f061819fab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0604009230 tx=0x7f060400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.479+0000 7f0612a00700 1 -- 172.21.15.63:0/1834488773 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604005040 con 0x7f0618109940 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.479+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06181a4810 con 0x7f0618109940 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.479+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06181ac0f0 con 0x7f0618109940 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.479+0000 7f0612a00700 1 -- 172.21.15.63:0/1834488773 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0604005690 con 0x7f0618109940 2024-04-25T18:46:21.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.479+0000 7f0612a00700 1 -- 172.21.15.63:0/1834488773 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060400b560 con 0x7f0618109940 2024-04-25T18:46:21.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.482+0000 7f0612a00700 1 -- 172.21.15.63:0/1834488773 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0604003680 con 0x7f0618109940 2024-04-25T18:46:21.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f0612a00700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 0x7f06000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 Processor -- start 2024-04-25T18:46:21.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f0612a00700 1 -- 172.21.15.63:0/1834488773 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f06040d11e0 con 0x7f0618109940 2024-04-25T18:46:21.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f061de00700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 0x7f06000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 -- start start 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c801030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56364c746e40 con 0x56364c7f33e0 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56364c6ed6e0 con 0x56364c7f3d00 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.484+0000 7f062b163b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56364c742e00 con 0x56364c7fec40 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57548/0 (socket says 172.21.15.63:57548) 2024-04-25T18:46:21.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42532/0 (socket says 172.21.15.63:42532) 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 -- 172.21.15.63:0/755071212 learned_addr learned my addr 172.21.15.63:0/755071212 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061e800700 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c801030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061de00700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 0x7f06000a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f060c009750 tx=0x7f060c011a50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 -- 172.21.15.63:0/755071212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 msgr2=0x56364c801030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c801030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 -- 172.21.15.63:0/755071212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 msgr2=0x56364c7f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.485+0000 7f061d400700 1 -- 172.21.15.63:0/755071212 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56364c6e3630 con 0x56364c7f3d00 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f061d400700 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06040cebc0 tx=0x7f0604009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c57adceaf49377fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f060b400700 1 -- 172.21.15.63:0/755071212 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060409dbc0 con 0x56364c7f3d00 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f060b400700 1 -- 172.21.15.63:0/755071212 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060409d4c0 con 0x56364c7f3d00 2024-04-25T18:46:21.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f060b400700 1 -- 172.21.15.63:0/755071212 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060409d720 con 0x56364c7f3d00 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f062b163b80 1 -- 172.21.15.63:0/755071212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 msgr2=0x56364c7fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.486+0000 7f062b163b80 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06040cebc0 tx=0x7f0604009260 comp rx=0 tx=0).stop 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 -- 172.21.15.63:0/755071212 shutdown_connections 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c801030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 --2- 172.21.15.63:0/755071212 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f3d00 0x56364c7fe700 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 -- 172.21.15.63:0/755071212 >> 172.21.15.63:0/755071212 conn(0x56364c722550 msgr2=0x56364c7efa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 -- 172.21.15.63:0/755071212 shutdown_connections 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 -- 172.21.15.63:0/755071212 wait complete. 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.487+0000 7f062b163b80 1 -- start start 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 Processor -- start 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c7fbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56364c801570 con 0x56364c7f33e0 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56364c700810 con 0x56364c7fec40 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56364c8016f0 con 0x56364c7f3d00 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f062b163b80 10 client.0 _ll_register_callbacks cb 0x56364c6df6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.488+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c7fbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42538/0 (socket says 172.21.15.63:42538) 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54510/0 (socket says 172.21.15.63:54510) 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 -- 172.21.15.63:0/1656927268 learned_addr learned my addr 172.21.15.63:0/1656927268 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 -- 172.21.15.63:0/1656927268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 msgr2=0x56364c7fb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 -- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 msgr2=0x56364c7fbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c7fbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 -- 172.21.15.63:0/1656927268 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56364c694880 con 0x56364c7f33e0 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061e800700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c7fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061d400700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f061de00700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f060c01eb50 tx=0x7f060c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddafd4f10ef02e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.489+0000 7f05ffe00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c01ee60 con 0x56364c7f33e0 2024-04-25T18:46:21.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f060c00fc40 con 0x56364c7f33e0 2024-04-25T18:46:21.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060409d4c0 con 0x56364c7f33e0 2024-04-25T18:46:21.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f060c017040 con 0x56364c7f33e0 2024-04-25T18:46:21.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f062b163b80 10 client.14898 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:21.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56364c6ddaa0 con 0x56364c7f33e0 2024-04-25T18:46:21.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.490+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060409d4c0 con 0x56364c7f33e0 2024-04-25T18:46:21.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f060c025030 con 0x56364c7f33e0 2024-04-25T18:46:21.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 10 client.14898 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 10 client.14898 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 20 client.14898 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c80e2d0 0x56364c80f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 4 client.14898 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56364c754630 con 0x56364c80e2d0 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.491+0000 7f061d400700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c80e2d0 0x56364c80f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.492+0000 7f061d400700 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c80e2d0 0x56364c80f740 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40fe6f1e7c6a0395 server_cookie=4fcc13887efdd768 in_seq=0 out_seq=0 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.492+0000 7f05ffe00700 10 client.14898 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.492+0000 7f05ffe00700 1 -- 172.21.15.63:0/1656927268 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56364c754630 con 0x56364c80e2d0 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.492+0000 7f05ffe00700 10 client.14898 handle_command_reply: tid=0 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.492+0000 7f062b163b80 1 client.14898 shutdown 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 msgr2=0x56364c7fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f060c01eb50 tx=0x7f060c0047c0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 shutdown_connections 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c80e2d0 0x56364c80f740 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c7fbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c7fe2e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 --2- 172.21.15.63:0/1656927268 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c7fb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 >> 172.21.15.63:0/1656927268 conn(0x56364c722550 msgr2=0x56364c7ef680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f05ffe00700 0 client.14898 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 shutdown_connections 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 1 -- 172.21.15.63:0/1656927268 wait complete. 2024-04-25T18:46:21.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.493+0000 7f062b163b80 20 client.14898 trim_cache size 0 max 16384 2024-04-25T18:46:21.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 Processor -- start 2024-04-25T18:46:21.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 -- start start 2024-04-25T18:46:21.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f3d00 0x56364c7efb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c7f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56364c80da80 con 0x56364c7f3d00 2024-04-25T18:46:21.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56364c80e150 con 0x56364c7f33e0 2024-04-25T18:46:21.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.516+0000 7f062b163b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56364c80e820 con 0x56364c7fec40 2024-04-25T18:46:21.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c7f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f3d00 0x56364c7efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57566/0 (socket says 172.21.15.63:57566) 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 -- 172.21.15.63:0/1798690564 learned_addr learned my addr 172.21.15.63:0/1798690564 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 -- 172.21.15.63:0/1798690564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 msgr2=0x56364c7f88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c7f88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 -- 172.21.15.63:0/1798690564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f3d00 msgr2=0x56364c7efb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f3d00 0x56364c7efb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 -- 172.21.15.63:0/1798690564 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56364c7f9b70 con 0x56364c7f33e0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f061de00700 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f060c003d70 tx=0x7f060c00f780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23cca99793f23450 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.517+0000 7f060aa00700 1 -- 172.21.15.63:0/1798690564 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c035070 con 0x56364c7f33e0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f060aa00700 1 -- 172.21.15.63:0/1798690564 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060c019050 con 0x56364c7f33e0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f060aa00700 1 -- 172.21.15.63:0/1798690564 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c01ee60 con 0x56364c7f33e0 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 -- 172.21.15.63:0/1798690564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 msgr2=0x56364c7fbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f060c003d70 tx=0x7f060c00f780 comp rx=0 tx=0).stop 2024-04-25T18:46:21.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 -- 172.21.15.63:0/1798690564 shutdown_connections 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7fec40 0x56364c7f88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7f33e0 0x56364c7fbad0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 --2- 172.21.15.63:0/1798690564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f3d00 0x56364c7efb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 -- 172.21.15.63:0/1798690564 >> 172.21.15.63:0/1798690564 conn(0x56364c722550 msgr2=0x56364c7fb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 -- 172.21.15.63:0/1798690564 shutdown_connections 2024-04-25T18:46:21.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.518+0000 7f062b163b80 1 -- 172.21.15.63:0/1798690564 wait complete. 2024-04-25T18:46:21.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.519+0000 7f062b163b80 1 -- start start 2024-04-25T18:46:21.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.519+0000 7f062b163b80 1 Processor -- start 2024-04-25T18:46:21.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c814cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c815210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56364c80f5c0 con 0x56364c7f33e0 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56364c80eef0 con 0x56364c7fec40 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56364c80fc90 con 0x56364c7f3d00 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f062b163b80 10 client.0 _ll_register_callbacks cb 0x56364c6df6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42552/0 (socket says 172.21.15.63:42552) 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 -- 172.21.15.63:0/1875977447 learned_addr learned my addr 172.21.15.63:0/1875977447 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061e800700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c815210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061d400700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c814cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 -- 172.21.15.63:0/1875977447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 msgr2=0x56364c814cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c814cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 -- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 msgr2=0x56364c815210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:21.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c815210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.520+0000 7f061de00700 1 -- 172.21.15.63:0/1875977447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56364c874660 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f061de00700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f060c0041c0 tx=0x7f060c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fba0c047fee7b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f05ffe00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c035070 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f060c01e270 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f060c019050 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f060c01ee60 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f062b163b80 10 client.14904 fetch_fsmap learned FSMap version 23 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.521+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56364c875900 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060c019050 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f060c043070 con 0x56364c7f33e0 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 10 client.14904 fetch_fsmap finished waiting for FSMap version 23 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 10 client.14904 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:46:21.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 20 client.14904 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c87a130 0x56364c878e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 4 client.14904 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:46:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "status"}) -- 0x56364c74e6d0 con 0x56364c87a130 2024-04-25T18:46:21.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.522+0000 7f061d400700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c87a130 0x56364c878e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f061d400700 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c87a130 0x56364c878e40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbfdfaab6eb399cf server_cookie=a56aae8f682c5933 in_seq=0 out_seq=0 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f05ffe00700 10 client.14904 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f05ffe00700 1 -- 172.21.15.63:0/1875977447 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56364c74e6d0 con 0x56364c87a130 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f05ffe00700 10 client.14904 handle_command_reply: tid=0 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 client.14904 shutdown 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 msgr2=0x56364c814790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f060c0041c0 tx=0x7f060c004760 comp rx=0 tx=0).stop 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 shutdown_connections 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56364c87a130 0x56364c878e40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56364c7fec40 0x56364c815210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.523+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56364c7f33e0 0x56364c814790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f05ffe00700 0 client.14904 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f062b163b80 1 --2- 172.21.15.63:0/1875977447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56364c7f3d00 0x56364c814cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 >> 172.21.15.63:0/1875977447 conn(0x56364c722550 msgr2=0x56364c7fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 shutdown_connections 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f062b163b80 1 -- 172.21.15.63:0/1875977447 wait complete. 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.524+0000 7f062b163b80 20 client.14904 trim_cache size 0 max 16384 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 0, 2024-04-25T18:46:21.528 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24533, 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 228.62897998099999, 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 23, 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 96, 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 94, 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 252.54534586099999 2024-04-25T18:46:21.529 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 msgr2=0x7f06000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 0x7f06000a6180 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f060c009750 tx=0x7f060c011a50 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 msgr2=0x7f061819fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f061819fab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0604009230 tx=0x7f060400fa00 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 shutdown_connections 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f06000a3cc0 0x7f06000a6180 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061810f030 0x7f06181a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0618108f80 0x7f061819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 --2- 172.21.15.63:0/1834488773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0618109940 0x7f061819fab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 >> 172.21.15.63:0/1834488773 conn(0x7f06180787d0 msgr2=0x7f0618079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 shutdown_connections 2024-04-25T18:46:21.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:21.525+0000 7f061f200700 1 -- 172.21.15.63:0/1834488773 wait complete. 2024-04-25T18:46:22.514 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T18:46:22.523 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T18:46:22.524 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T18:46:22.524 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T18:46:22.524 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-25T18:46:22.525 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T18:46:22.525 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:46:22.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-25T18:46:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:22 smithi114 ceph-mon[32988]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:22.605 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:22 smithi063 ceph-mon[28006]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:22 smithi070 ceph-mon[32820]: pgmap v460: 97 pgs: 97 active+clean; 639 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.739+0000 7f9f35800640 1 -- 172.21.15.63:0/3234166640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30102850 msgr2=0x7f9f30104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.739+0000 7f9f35800640 1 --2- 172.21.15.63:0/3234166640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30102850 0x7f9f30104c40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009ca0 tx=0x7f9f2002a8c0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 -- 172.21.15.63:0/3234166640 shutdown_connections 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 --2- 172.21.15.63:0/3234166640 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30105180 0x7f9f30107570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 --2- 172.21.15.63:0/3234166640 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30107ab0 0x7f9f30109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 --2- 172.21.15.63:0/3234166640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30102850 0x7f9f30104c40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 -- 172.21.15.63:0/3234166640 >> 172.21.15.63:0/3234166640 conn(0x7f9f300fc240 msgr2=0x7f9f300fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:22.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 -- 172.21.15.63:0/3234166640 shutdown_connections 2024-04-25T18:46:22.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.740+0000 7f9f35800640 1 -- 172.21.15.63:0/3234166640 wait complete. 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 Processor -- start 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 -- start start 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30102850 0x7f9f3019a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 0x7f9f3019a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 0x7f9f301a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3010d970 con 0x7f9f30107ab0 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3010d7f0 con 0x7f9f30102850 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.741+0000 7f9f35800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3010daf0 con 0x7f9f30105180 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f34e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 0x7f9f301a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30102850 0x7f9f3019a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:22.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f34e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 0x7f9f301a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42572/0 (socket says 172.21.15.63:42572) 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f34e00640 1 -- 172.21.15.63:0/453892726 learned_addr learned my addr 172.21.15.63:0/453892726 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 0x7f9f3019a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 -- 172.21.15.63:0/453892726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30102850 msgr2=0x7f9f3019a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30102850 0x7f9f3019a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 -- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 msgr2=0x7f9f301a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 0x7f9f301a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 -- 172.21.15.63:0/453892726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f301a1ea0 con 0x7f9f30105180 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.742+0000 7f9f2f400640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 0x7f9f3019a7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800e9c0 tx=0x7f9f1800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.743+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1800cd80 con 0x7f9f30105180 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.743+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f180035f0 con 0x7f9f30105180 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.743+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f301a2190 con 0x7f9f30105180 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.743+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f18010690 con 0x7f9f30105180 2024-04-25T18:46:22.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.743+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f301a2730 con 0x7f9f30105180 2024-04-25T18:46:22.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.744+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f30109600 con 0x7f9f30105180 2024-04-25T18:46:22.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.747+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9f180037a0 con 0x7f9f30105180 2024-04-25T18:46:22.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.748+0000 7f9f2cc00640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 0x7f9f140a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:22.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.748+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f9f180d0710 con 0x7f9f30105180 2024-04-25T18:46:22.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.748+0000 7f9f2fe00640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 0x7f9f140a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:22.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.749+0000 7f9f2fe00640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 0x7f9f140a60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f20005e30 tx=0x7f9f20005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:22.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:22.752+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9f1809d060 con 0x7f9f30105180 2024-04-25T18:46:23.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.014+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f9f30062b50 con 0x7f9f30105180 2024-04-25T18:46:23.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.326+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f9f18009310 con 0x7f9f30105180 2024-04-25T18:46:23.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.327+0000 7f9f2cc00640 1 -- 172.21.15.63:0/453892726 <== mon.1 v2:172.21.15.114:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f9f18078530 con 0x7f9f30105180 2024-04-25T18:46:23.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 msgr2=0x7f9f140a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 0x7f9f140a60e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f20005e30 tx=0x7f9f20005f50 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 msgr2=0x7f9f3019a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 0x7f9f3019a7c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800e9c0 tx=0x7f9f1800ee80 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 shutdown_connections 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f9f140a3c20 0x7f9f140a60e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f30107ab0 0x7f9f301a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9f30105180 0x7f9f3019a7c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 --2- 172.21.15.63:0/453892726 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9f30102850 0x7f9f3019a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 >> 172.21.15.63:0/453892726 conn(0x7f9f300fc240 msgr2=0x7f9f30102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.332+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 shutdown_connections 2024-04-25T18:46:23.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.333+0000 7f9f35800640 1 -- 172.21.15.63:0/453892726 wait complete. 2024-04-25T18:46:23.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-25T18:46:23.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 -- 172.21.15.63:0/659806926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2688105d80 msgr2=0x7f268810c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 --2- 172.21.15.63:0/659806926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2688105d80 0x7f268810c610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2680009900 tx=0x7f268002a410 comp rx=0 tx=0).stop 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 -- 172.21.15.63:0/659806926 shutdown_connections 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 --2- 172.21.15.63:0/659806926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2688105d80 0x7f268810c610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 --2- 172.21.15.63:0/659806926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f26881053c0 0x7f2688105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 --2- 172.21.15.63:0/659806926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 0x7f26881045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 -- 172.21.15.63:0/659806926 >> 172.21.15.63:0/659806926 conn(0x7f26880ff740 msgr2=0x7f2688101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 -- 172.21.15.63:0/659806926 shutdown_connections 2024-04-25T18:46:23.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.539+0000 7f268cc00640 1 -- 172.21.15.63:0/659806926 wait complete. 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.540+0000 7f268cc00640 1 Processor -- start 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.540+0000 7f268cc00640 1 -- start start 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 0x7f268819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2688105d80 0x7f26881a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f268810fd60 con 0x7f26881041d0 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f268810fbe0 con 0x7f26881053c0 2024-04-25T18:46:23.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f268cc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f268810fee0 con 0x7f2688105d80 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57608/0 (socket says 172.21.15.63:57608) 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 -- 172.21.15.63:0/1412332000 learned_addr learned my addr 172.21.15.63:0/1412332000 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f2685e00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 0x7f268819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f2686800640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2688105d80 0x7f26881a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2688105d80 msgr2=0x7f26881a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2688105d80 0x7f26881a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 msgr2=0x7f268819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 0x7f268819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.541+0000 7f267d400640 1 -- 172.21.15.63:0/1412332000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26881a1ef0 con 0x7f26881053c0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f267d400640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f267000eb90 tx=0x7f267000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670004d20 con 0x7f26881053c0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267000ee90 con 0x7f26881053c0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670010670 con 0x7f26881053c0 2024-04-25T18:46:23.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26881a21e0 con 0x7f26881053c0 2024-04-25T18:46:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.542+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26881a26e0 con 0x7f26881053c0 2024-04-25T18:46:23.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.543+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268810b4f0 con 0x7f26881053c0 2024-04-25T18:46:23.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.545+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f2670005020 con 0x7f26881053c0 2024-04-25T18:46:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.547+0000 7f267f400640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 0x7f266c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.547+0000 7f2685e00640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 msgr2=0x7f266c0a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:23.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.548+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7f26700d0a90 con 0x7f26881053c0 2024-04-25T18:46:23.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.548+0000 7f2685e00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 0x7f266c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:46:23.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.555+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f267009d3e0 con 0x7f26881053c0 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/453892726' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: ignoring --setuser ceph since I am not root 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: ignoring --setgroup ceph since I am not root 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-mgr[34850]: -- 172.21.15.114:0/881182092 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cab6f025a0 con 0x55cab6ed7000 2024-04-25T18:46:23.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:23.497+0000 7f943f7b1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:46:23.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/453892726' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setuser ceph since I am not root 2024-04-25T18:46:23.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: ignoring --setgroup ceph since I am not root 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/453892726' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: ignoring --setuser ceph since I am not root 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: ignoring --setgroup ceph since I am not root 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-mgr[34212]: -- 172.21.15.70:0/1571311385 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e7fdef85a0 con 0x55e7fdeccc00 2024-04-25T18:46:23.684 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:23.495+0000 7fa561130200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:46:23.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.749+0000 7f2685e00640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 msgr2=0x7f266c0a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:23.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.749+0000 7f2685e00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 0x7f266c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:46:23.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.765+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f268802d5c0 con 0x7f26881053c0 2024-04-25T18:46:23.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.774+0000 7f267f400640 1 -- 172.21.15.63:0/1412332000 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) ==== 138+0+0 (secure 0 0 0) 0x7f267009cce0 con 0x7f26881053c0 2024-04-25T18:46:23.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 msgr2=0x7f266c0a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 0x7f266c0a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 msgr2=0x7f268819a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f267000eb90 tx=0x7f267000c6a0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 shutdown_connections 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7f266c0a3d20 0x7f266c0a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26881053c0 0x7f268819a810 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2688105d80 0x7f26881a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 --2- 172.21.15.63:0/1412332000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26881041d0 0x7f268819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.776+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 >> 172.21.15.63:0/1412332000 conn(0x7f26880ff740 msgr2=0x7f26881086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.777+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 shutdown_connections 2024-04-25T18:46:23.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.777+0000 7f268cc00640 1 -- 172.21.15.63:0/1412332000 wait complete. 2024-04-25T18:46:23.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-25T18:46:23.908 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:23.646+0000 7ff4a4ed2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T18:46:23.953 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:23.682+0000 7fa561130200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:23.953 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:23.787+0000 7fa561130200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 -- 172.21.15.63:0/2549347472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540101de0 msgr2=0x7fc540102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 --2- 172.21.15.63:0/2549347472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540101de0 0x7fc540102200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc530009900 tx=0x7fc53002a410 comp rx=0 tx=0).stop 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 -- 172.21.15.63:0/2549347472 shutdown_connections 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 --2- 172.21.15.63:0/2549347472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540103990 0x7fc54010a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 --2- 172.21.15.63:0/2549347472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540102fd0 0x7fc540103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 --2- 172.21.15.63:0/2549347472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540101de0 0x7fc540102200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.981+0000 7fc545600640 1 -- 172.21.15.63:0/2549347472 >> 172.21.15.63:0/2549347472 conn(0x7fc5400fd350 msgr2=0x7fc5400ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.982+0000 7fc545600640 1 -- 172.21.15.63:0/2549347472 shutdown_connections 2024-04-25T18:46:23.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.982+0000 7fc545600640 1 -- 172.21.15.63:0/2549347472 wait complete. 2024-04-25T18:46:23.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.982+0000 7fc545600640 1 Processor -- start 2024-04-25T18:46:23.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 -- start start 2024-04-25T18:46:23.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 0x7fc54019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540103990 0x7fc5401a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc54010d930 con 0x7fc540103990 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc54010d7b0 con 0x7fc540102fd0 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc545600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc54010dab0 con 0x7fc540101de0 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc53fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc53fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54584/0 (socket says 172.21.15.63:54584) 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc53fe00640 1 -- 172.21.15.63:0/1867513773 learned_addr learned my addr 172.21.15.63:0/1867513773 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc53f400640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 0x7fc54019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.983+0000 7fc544c00640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540103990 0x7fc5401a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 msgr2=0x7fc54019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 0x7fc54019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540103990 msgr2=0x7fc5401a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:23.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540103990 0x7fc5401a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 -- 172.21.15.63:0/1867513773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5401a1e60 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53f400640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 0x7fc54019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53fe00640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc530031500 tx=0x7fc530003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc53003f070 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5401a20f0 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc530031c50 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc530035030 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.984+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5401a2630 con 0x7fc540101de0 2024-04-25T18:46:23.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.986+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5401a2890 con 0x7fc540101de0 2024-04-25T18:46:23.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.987+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fc530042070 con 0x7fc540101de0 2024-04-25T18:46:23.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.988+0000 7fc53cc00640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 0x7fc5240a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:23.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.989+0000 7fc53f400640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 msgr2=0x7fc5240a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:23.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.989+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fc5300f0bf0 con 0x7fc540101de0 2024-04-25T18:46:23.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.989+0000 7fc53f400640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 0x7fc5240a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:46:23.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:23.992+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5300bcef0 con 0x7fc540101de0 2024-04-25T18:46:24.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:23.679+0000 7f943f7b1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:24.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:23.783+0000 7f943f7b1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:46:24.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.190+0000 7fc53f400640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 msgr2=0x7fc5240a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:24.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.190+0000 7fc53f400640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 0x7fc5240a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:46:24.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.216+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fc5401a2c10 con 0x7fc540101de0 2024-04-25T18:46:24.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.225+0000 7fc53cc00640 1 -- 172.21.15.63:0/1867513773 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) ==== 133+0+0 (secure 0 0 0) 0x7fc5300c11f0 con 0x7fc540101de0 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.227+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 msgr2=0x7fc5240a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 0x7fc5240a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 msgr2=0x7fc54019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc530031500 tx=0x7fc530003c10 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 shutdown_connections 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fc5240a3bf0 0x7fc5240a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc540102fd0 0x7fc54019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc540103990 0x7fc5401a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 --2- 172.21.15.63:0/1867513773 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc540101de0 0x7fc54019a240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:24.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 >> 172.21.15.63:0/1867513773 conn(0x7fc5400fd350 msgr2=0x7fc540074180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:24.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 shutdown_connections 2024-04-25T18:46:24.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:24.228+0000 7fc545600640 1 -- 172.21.15.63:0/1867513773 wait complete. 2024-04-25T18:46:24.245 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:23.907+0000 7ff4a4ed2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:24.245 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:24.069+0000 7ff4a4ed2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T18:46:24.248 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T18:46:24.269 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T18:46:24.269 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T18:46:24.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:24.269 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-04-25T18:46:24.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:24.323 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link delete ceph-brx 2024-04-25T18:46:24.404 INFO:teuthology.orchestra.run.smithi063.stderr:Cannot find device "ceph-brx" 2024-04-25T18:46:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T18:46:24.408 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-25T18:46:24.408 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T18:46:24.408 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T18:46:24.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T18:46:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T18:46:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-mon[32820]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-04-25T18:46:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.453 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:24.095+0000 7fa561130200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:46:24.454 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:24.180+0000 7fa561130200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:46:24.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T18:46:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-mon[32988]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-04-25T18:46:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:24.090+0000 7f943f7b1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:46:24.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:24.175+0000 7f943f7b1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-mon[28006]: mgrmap e19: x(active, since 12m), standbys: z, y 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:24.484+0000 7ff4a4ed2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T18:46:24.606 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:24.604+0000 7ff4a4ed2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T18:46:24.844 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:25.688 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:25.350+0000 7f943f7b1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:46:25.688 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:25.519+0000 7f943f7b1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:25.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:25.364+0000 7fa561130200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:46:25.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:25.533+0000 7fa561130200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:25.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:25.702+0000 7fa561130200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:46:25.977 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:25.687+0000 7f943f7b1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:46:26.269 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:25.975+0000 7f943f7b1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:46:26.269 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:26.067+0000 7f943f7b1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:46:26.283 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:25.990+0000 7fa561130200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:46:26.283 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:26.082+0000 7fa561130200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:46:26.283 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:26.282+0000 7fa561130200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:46:26.477 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:26.229+0000 7ff4a4ed2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T18:46:26.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:26.267+0000 7f943f7b1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:46:26.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:26.459+0000 7f943f7b1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:26.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:26.474+0000 7fa561130200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:26.737 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:26.475+0000 7ff4a4ed2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:26.737 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:26.699+0000 7ff4a4ed2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 -- 172.21.15.63:0/350097890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0107940 msgr2=0x7fa6f0107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 --2- 172.21.15.63:0/350097890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0107940 0x7fa6f0107d20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009990 tx=0x7fa6e0009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 -- 172.21.15.63:0/350097890 shutdown_connections 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 --2- 172.21.15.63:0/350097890 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f0107940 0x7fa6f0107d20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 --2- 172.21.15.63:0/350097890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f01034b0 0x7fa6f010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 --2- 172.21.15.63:0/350097890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0102af0 0x7fa6f0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 -- 172.21.15.63:0/350097890 >> 172.21.15.63:0/350097890 conn(0x7fa6f0079ba0 msgr2=0x7fa6f0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 -- 172.21.15.63:0/350097890 shutdown_connections 2024-04-25T18:46:27.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.040+0000 7fa6f7200700 1 -- 172.21.15.63:0/350097890 wait complete. 2024-04-25T18:46:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 Processor -- start 2024-04-25T18:46:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 -- start start 2024-04-25T18:46:27.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 0x7fa6f019b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:27.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 0x7fa6f019b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:27.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 0x7fa6f019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:27.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0112ca0 con 0x7fa6f0102af0 2024-04-25T18:46:27.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0112b20 con 0x7fa6f0107940 2024-04-25T18:46:27.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.042+0000 7fa6f7200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0112e20 con 0x7fa6f01034b0 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6efe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 0x7fa6f019b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6f5200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 0x7fa6f019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 0x7fa6f019b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6f5200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 0x7fa6f019fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57652/0 (socket says 172.21.15.63:57652) 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6f5200700 1 -- 172.21.15.63:0/3043409707 learned_addr learned my addr 172.21.15.63:0/3043409707 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 msgr2=0x7fa6f019fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:27.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 0x7fa6f019fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 msgr2=0x7fa6f019b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 0x7fa6f019b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6ef400700 1 -- 172.21.15.63:0/3043409707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f01a03e0 con 0x7fa6f01034b0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.043+0000 7fa6efe00700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 0x7fa6f019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6ef400700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 0x7fa6f019b910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e9f0 tx=0x7fa6dc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc004220 con 0x7fa6f01034b0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc0043c0 con 0x7fa6f01034b0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f01a06d0 con 0x7fa6f01034b0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc005020 con 0x7fa6f01034b0 2024-04-25T18:46:27.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.044+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f01a7f50 con 0x7fa6f01034b0 2024-04-25T18:46:27.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.045+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f0069730 con 0x7fa6f01034b0 2024-04-25T18:46:27.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.049+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fa6dc004750 con 0x7fa6f01034b0 2024-04-25T18:46:27.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.050+0000 7fa6ecc00700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 0x7fa6d40a6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:27.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.050+0000 7fa6efe00700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 msgr2=0x7fa6d40a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:27.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.050+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fa6dc0d05c0 con 0x7fa6f01034b0 2024-04-25T18:46:27.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.050+0000 7fa6efe00700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 0x7fa6d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:46:27.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.051+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6dc042e20 con 0x7fa6f01034b0 2024-04-25T18:46:27.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.251+0000 7fa6efe00700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 msgr2=0x7fa6d40a6150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:27.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.251+0000 7fa6efe00700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 0x7fa6d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:46:27.316 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:27.052+0000 7f943f7b1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:46:27.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.332+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa6f019c730 con 0x7fa6f01034b0 2024-04-25T18:46:27.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.333+0000 7fa6ecc00700 1 -- 172.21.15.63:0/3043409707 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fa6dc09cf70 con 0x7fa6f01034b0 2024-04-25T18:46:27.334 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:27.111+0000 7ff4a4ed2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T18:46:27.335 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:27.246+0000 7ff4a4ed2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T18:46:27.335 INFO:teuthology.orchestra.run.smithi063.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 msgr2=0x7fa6d40a6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 0x7fa6d40a6150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 msgr2=0x7fa6f019b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 0x7fa6f019b910 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e9f0 tx=0x7fa6dc00ed00 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 shutdown_connections 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa6d40a3c90 0x7fa6d40a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6f0107940 0x7fa6f019fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6f0102af0 0x7fa6f019b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 --2- 172.21.15.63:0/3043409707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6f01034b0 0x7fa6f019b910 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 >> 172.21.15.63:0/3043409707 conn(0x7fa6f0079ba0 msgr2=0x7fa6f0100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.336+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 shutdown_connections 2024-04-25T18:46:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:27.337+0000 7fa6f7200700 1 -- 172.21.15.63:0/3043409707 wait complete. 2024-04-25T18:46:27.386 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:27.068+0000 7fa561130200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:46:27.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:27 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3043409707' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T18:46:27.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:27.512+0000 7ff4a4ed2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T18:46:27.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:27 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3043409707' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T18:46:27.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:27.488+0000 7fa561130200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:46:27.704 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:27.583+0000 7fa561130200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:46:27.767 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:27 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3043409707' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T18:46:27.767 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:27.472+0000 7f943f7b1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:46:27.767 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:27.566+0000 7f943f7b1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:46:27.767 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:27.765+0000 7f943f7b1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:46:28.119 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:27.780+0000 7ff4a4ed2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T18:46:28.176 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T18:46:28.176 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi063.front.sepia.ceph.com 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T18:46:28.177 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T18:46:28.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:28.177 DEBUG:teuthology.orchestra.run.smithi063:> ip addr 2024-04-25T18:46:28.203 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:27.783+0000 7fa561130200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:46:28.231 INFO:teuthology.orchestra.run.smithi063.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T18:46:28.231 INFO:teuthology.orchestra.run.smithi063.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T18:46:28.231 INFO:teuthology.orchestra.run.smithi063.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T18:46:28.231 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-04-25T18:46:28.231 INFO:teuthology.orchestra.run.smithi063.stdout: inet6 ::1/128 scope host 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:a8:b4:76 brd ff:ff:ff:ff:ff:ff 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: altname enp7s0f0 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:a8:b4:77 brd ff:ff:ff:ff:ff:ff 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: altname enp7s0f1 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:b6:f8:d6 brd ff:ff:ff:ff:ff:ff 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: link/ether 0c:c4:7a:b6:f8:d7 brd ff:ff:ff:ff:ff:ff 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: inet 172.21.15.63/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft 41772sec preferred_lft 41772sec 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link 2024-04-25T18:46:28.232 INFO:teuthology.orchestra.run.smithi063.stdout: valid_lft forever preferred_lft forever 2024-04-25T18:46:28.233 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link add name ceph-brx type bridge 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip addr flush dev ceph-brx 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set ceph-brx up 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T18:46:28.234 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-04-25T18:46:28.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:28.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.282+0000 7f943f7b1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:46:28.567 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.374+0000 7f943f7b1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:46:28.638 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.303+0000 7fa561130200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:46:28.638 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.395+0000 7fa561130200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:46:28.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:28.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:28.702 DEBUG:teuthology.orchestra.run.smithi063:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T18:46:28.766 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-04-25T18:46:28.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:28.769 DEBUG:teuthology.orchestra.run.smithi063:> route 2024-04-25T18:46:28.823 INFO:teuthology.orchestra.run.smithi063.stdout:Kernel IP routing table 2024-04-25T18:46:28.823 INFO:teuthology.orchestra.run.smithi063.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T18:46:28.823 INFO:teuthology.orchestra.run.smithi063.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T18:46:28.823 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T18:46:28.823 INFO:teuthology.orchestra.run.smithi063.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T18:46:28.824 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-04-25T18:46:28.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:28.614+0000 7ff4a4ed2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T18:46:28.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:28.953 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.637+0000 7fa561130200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:46:28.953 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.771+0000 7fa561130200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:46:28.953 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.865+0000 7fa561130200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:46:28.954 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.612+0000 7f943f7b1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:46:28.954 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.744+0000 7f943f7b1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:46:28.954 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.837+0000 7f943f7b1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:46:28.954 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:28.952+0000 7f943f7b1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:29.285 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:28.985+0000 7fa561130200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:29.566 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:29.250+0000 7f943f7b1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:46:29.573 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:29.241+0000 7ff4a4ed2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T18:46:29.573 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:29.381+0000 7ff4a4ed2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T18:46:29.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:29.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:29.585 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-04-25T18:46:29.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:29.631 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list-id 2024-04-25T18:46:29.703 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:29.284+0000 7fa561130200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:46:29.712 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T18:46:29.712 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-04-25T18:46:29.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T18:46:29.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T18:46:29.712 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-04-25T18:46:29.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:29.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:29.689+0000 7ff4a4ed2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T18:46:29.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:29.937 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T18:46:29.937 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-04-25T18:46:30.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:30.139 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:29.884+0000 7fa561130200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:46:30.140 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y[34208]: 2024-04-25T18:46:30.101+0000 7fa561130200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:46:30.140 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:29.850+0000 7f943f7b1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:46:30.141 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z[34846]: 2024-04-25T18:46:30.063+0000 7f943f7b1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:46:30.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:30.268 DEBUG:teuthology.orchestra.run.smithi063:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T18:46:30.268 DEBUG:teuthology.orchestra.run.smithi063:> set -e 2024-04-25T18:46:30.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set brx.0 up 2024-04-25T18:46:30.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T18:46:30.268 DEBUG:teuthology.orchestra.run.smithi063:> ') 2024-04-25T18:46:30.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: Standby manager daemon z restarted 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: Standby manager daemon z started 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: Standby manager daemon y restarted 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: Standby manager daemon y started 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:46:30.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-mon[28006]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: Standby manager daemon z restarted 2024-04-25T18:46:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: Standby manager daemon z started 2024-04-25T18:46:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:46:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: Standby manager daemon y restarted 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: Standby manager daemon y started 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:46:30.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:30 smithi070 ceph-mon[32820]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:30.476 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T18:46:30.476 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T18:46:30.519 INFO:teuthology.orchestra.run.smithi063.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T18:46:30.519 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T18:46:30.519 DEBUG:teuthology.orchestra.run.smithi063:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: Standby manager daemon z restarted 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: Standby manager daemon z started 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.114:0/2633877684' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: Standby manager daemon y restarted 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: Standby manager daemon y started 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T18:46:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T18:46:30.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T18:46:30.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:30 smithi114 ceph-mon[32988]: from='mgr.? 172.21.15.70:0/1665579042' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T18:46:30.609 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T18:46:30.609 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T18:46:30.609 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/fb5cabf5-5965-43c6-9259-1d222383970f 2024-04-25T18:46:30.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:30.609 DEBUG:teuthology.orchestra.run.smithi063:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/fb5cabf5-5965-43c6-9259-1d222383970f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-25T18:46:30.715 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:30.399+0000 7ff4a4ed2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T18:46:30.715 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:30.535+0000 7ff4a4ed2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T18:46:30.796 INFO:teuthology.orchestra.run.smithi063.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-25T18:46:30.796 INFO:teuthology.orchestra.run.smithi063.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-25T18:46:30.973 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:30.887+0000 7ff4a4ed2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T18:46:31.375 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:31 smithi063 ceph-mon[28006]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-25T18:46:31.375 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:31.076+0000 7ff4a4ed2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T18:46:31.375 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:31.213+0000 7ff4a4ed2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T18:46:31.375 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:31.372+0000 7ff4a4ed2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T18:46:31.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:31 smithi070 ceph-mon[32820]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-25T18:46:31.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:31 smithi114 ceph-mon[32988]: mgrmap e20: x(active, since 12m), standbys: y, z 2024-04-25T18:46:31.591 INFO:teuthology.orchestra.run.smithi063.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-04-25T18:46:31.591 INFO:teuthology.orchestra.run.smithi063.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-04-25T18:46:31.591 INFO:teuthology.orchestra.run.smithi063.stdout:mount.ceph: resolved to: "172.21.15.63:3300,172.21.15.70:3300,172.21.15.114:3300" 2024-04-25T18:46:31.591 INFO:teuthology.orchestra.run.smithi063.stdout:mount.ceph: trying mount with new device syntax: 0@e90c993c-0331-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/fb5cabf5-5965-43c6-9259-1d222383970f 2024-04-25T18:46:31.591 INFO:teuthology.orchestra.run.smithi063.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.63:3300/172.21.15.70:3300/172.21.15.114:3300" will pass to kernel 2024-04-25T18:46:31.594 INFO:teuthology.orchestra.run.smithi063.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T18:46:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: You just mounted a file system that supports labels which does not 2024-04-25T18:46:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T18:46:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T18:46:31.594 INFO:teuthology.orchestra.run.smithi063.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T18:46:31.597 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T18:46:31.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T18:46:31.598 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T18:46:31.681 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T18:46:31.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T18:46:31.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:31.803 INFO:teuthology.orchestra.run.smithi063.stdout:client25076 2024-04-25T18:46:31.803 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:46:31.803 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T18:46:31.859 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T18:46:31.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:31 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T18:46:31.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:31.973 INFO:teuthology.orchestra.run.smithi063.stdout:client25076 2024-04-25T18:46:31.973 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T18:46:32.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T18:46:32.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:32.089 INFO:teuthology.orchestra.run.smithi063.stdout:client25076 2024-04-25T18:46:32.089 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/e90c993c-0331-11ef-bc93-c7b262605968.client25076/status') 2024-04-25T18:46:32.127 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:31.761+0000 7ff4a4ed2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T18:46:32.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T18:46:32.220 INFO:teuthology.orchestra.run.smithi063.stdout:instance: client.25076 (3)172.21.15.63:0/2522287616 2024-04-25T18:46:32.220 INFO:teuthology.orchestra.run.smithi063.stdout:blocklisted: false 2024-04-25T18:46:32.221 INFO:teuthology.orchestra.run.smithi063.stderr:0+1 records in 2024-04-25T18:46:32.221 INFO:teuthology.orchestra.run.smithi063.stderr:0+1 records out 2024-04-25T18:46:32.221 INFO:teuthology.orchestra.run.smithi063.stderr:71 bytes copied, 8.1552e-05 s, 871 kB/s 2024-04-25T18:46:32.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T18:46:32.232 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T18:46:32.241 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T18:46:32.241 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:46:32.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T18:46:32.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 -- 172.21.15.63:0/1042552144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa041053c0 msgr2=0x7faa04105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:32.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 --2- 172.21.15.63:0/1042552144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa041053c0 0x7faa04105840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec009900 tx=0x7fa9ec02a410 comp rx=0 tx=0).stop 2024-04-25T18:46:32.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 -- 172.21.15.63:0/1042552144 shutdown_connections 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 --2- 172.21.15.63:0/1042552144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa041053c0 0x7faa04105840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 --2- 172.21.15.63:0/1042552144 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa04105d80 0x7faa0410c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 --2- 172.21.15.63:0/1042552144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041041d0 0x7faa041045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 -- 172.21.15.63:0/1042552144 >> 172.21.15.63:0/1042552144 conn(0x7faa040ff740 msgr2=0x7faa04101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 -- 172.21.15.63:0/1042552144 shutdown_connections 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.464+0000 7faa0a600640 1 -- 172.21.15.63:0/1042552144 wait complete. 2024-04-25T18:46:32.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 Processor -- start 2024-04-25T18:46:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 -- start start 2024-04-25T18:46:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 0x7faa0419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 0x7faa041a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa0410fd60 con 0x7faa041053c0 2024-04-25T18:46:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa0410fbe0 con 0x7faa04105d80 2024-04-25T18:46:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.466+0000 7faa0a600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faa0410fee0 con 0x7faa041041d0 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43874/0 (socket says 172.21.15.63:43874) 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 -- 172.21.15.63:0/4115947392 learned_addr learned my addr 172.21.15.63:0/4115947392 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7faa03400640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 0x7faa0419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7faa03e00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 0x7faa041a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 msgr2=0x7faa0419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 0x7faa0419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 msgr2=0x7faa041a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 0x7faa041a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 -- 172.21.15.63:0/4115947392 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa041a1f50 con 0x7faa041053c0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7faa03400640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 0x7faa0419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7faa03e00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 0x7faa041a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.467+0000 7fa9fbe00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec005ab0 tx=0x7fa9ec003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.468+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec03f070 con 0x7faa041053c0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.468+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9ec031c50 con 0x7faa041053c0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.468+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa041a21e0 con 0x7faa041053c0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.468+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec035030 con 0x7faa041053c0 2024-04-25T18:46:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.468+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa041a26d0 con 0x7faa041053c0 2024-04-25T18:46:32.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.469+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0410b4f0 con 0x7faa041053c0 2024-04-25T18:46:32.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.471+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 20) ==== 138663+0+0 (secure 0 0 0) 0x7fa9ec042070 con 0x7faa041053c0 2024-04-25T18:46:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.473+0000 7faa00c00640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 0x7fa9e80a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.473+0000 7faa03400640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 msgr2=0x7fa9e80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.473+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8285+0+0 (secure 0 0 0) 0x7fa9ec0f0d80 con 0x7faa041053c0 2024-04-25T18:46:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.473+0000 7faa03400640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 0x7fa9e80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T18:46:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.477+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9ec0bd6d0 con 0x7faa041053c0 2024-04-25T18:46:32.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.674+0000 7faa03400640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 msgr2=0x7fa9e80a6160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3659595029 2024-04-25T18:46:32.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.674+0000 7faa03400640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 0x7fa9e80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T18:46:32.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.729+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7faa04062b50 con 0x7faa041053c0 2024-04-25T18:46:32.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.783+0000 7faa00c00640 1 -- 172.21.15.63:0/4115947392 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7fa9ec0bcfd0 con 0x7faa041053c0 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 msgr2=0x7fa9e80a6160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 0x7fa9e80a6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 msgr2=0x7faa0419a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec005ab0 tx=0x7fa9ec003b90 comp rx=0 tx=0).stop 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 shutdown_connections 2024-04-25T18:46:32.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:6800/3659595029,v1:172.21.15.63:6801/3659595029] conn(0x7fa9e80a3ca0 0x7fa9e80a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa04105d80 0x7faa041a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa041053c0 0x7faa0419a870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 --2- 172.21.15.63:0/4115947392 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faa041041d0 0x7faa0419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:32.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.787+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 >> 172.21.15.63:0/4115947392 conn(0x7faa040ff740 msgr2=0x7faa041086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:32.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.788+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 shutdown_connections 2024-04-25T18:46:32.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:32.788+0000 7faa0a600640 1 -- 172.21.15.63:0/4115947392 wait complete. 2024-04-25T18:46:32.808 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T18:46:32.818 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T18:46:32.852 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T18:46:32.852 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:46:32.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-25T18:46:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:32 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T18:46:32.877 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:32.570+0000 7ff4a4ed2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T18:46:33.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 -- 172.21.15.63:0/2835794662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48100640 msgr2=0x7f8f48102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:33.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 --2- 172.21.15.63:0/2835794662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48100640 0x7f8f48102a70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009860 tx=0x7f8f3802a340 comp rx=0 tx=0).stop 2024-04-25T18:46:33.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 -- 172.21.15.63:0/2835794662 shutdown_connections 2024-04-25T18:46:33.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 --2- 172.21.15.63:0/2835794662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48105960 0x7f8f48105de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:33.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 --2- 172.21.15.63:0/2835794662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48102fb0 0x7f8f48105420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 --2- 172.21.15.63:0/2835794662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48100640 0x7f8f48102a70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 -- 172.21.15.63:0/2835794662 >> 172.21.15.63:0/2835794662 conn(0x7f8f480fc240 msgr2=0x7f8f480fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 -- 172.21.15.63:0/2835794662 shutdown_connections 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.055+0000 7f8f4d600640 1 -- 172.21.15.63:0/2835794662 wait complete. 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.056+0000 7f8f4d600640 1 Processor -- start 2024-04-25T18:46:33.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.056+0000 7f8f4d600640 1 -- start start 2024-04-25T18:46:33.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:33.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 0x7f8f4819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48105960 0x7f8f481a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8f48113ef0 con 0x7f8f48105960 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f48113d70 con 0x7f8f48102fb0 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f48114070 con 0x7f8f48100640 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f47e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f47e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49998/0 (socket says 172.21.15.63:49998) 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f47e00640 1 -- 172.21.15.63:0/3754348404 learned_addr learned my addr 172.21.15.63:0/3754348404 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f47400640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 0x7f8f4819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:33.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.057+0000 7f8f4cc00640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48105960 0x7f8f481a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 -- 172.21.15.63:0/3754348404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 msgr2=0x7f8f4819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 -- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48105960 msgr2=0x7f8f481a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48105960 0x7f8f481a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 -- 172.21.15.63:0/3754348404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f481a61f0 con 0x7f8f48102fb0 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47e00640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f47400640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 0x7f8f4819eb10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8f34005fd0 tx=0x7f8f3400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3400cd00 con 0x7f8f48102fb0 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f481a64e0 con 0x7f8f48102fb0 2024-04-25T18:46:33.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.058+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f340035f0 con 0x7f8f48102fb0 2024-04-25T18:46:33.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.059+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f34010690 con 0x7f8f48102fb0 2024-04-25T18:46:33.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.059+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f481a6a80 con 0x7f8f48102fb0 2024-04-25T18:46:33.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.060+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f480fdd30 con 0x7f8f48102fb0 2024-04-25T18:46:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:32 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T18:46:33.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.065+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f8f34003a80 con 0x7f8f48102fb0 2024-04-25T18:46:33.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.066+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f8f340d0130 con 0x7f8f48102fb0 2024-04-25T18:46:33.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.070+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f34009510 con 0x7f8f48102fb0 2024-04-25T18:46:33.116 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:32 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T18:46:33.224 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:32.877+0000 7ff4a4ed2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T18:46:33.224 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.222+0000 7ff3f9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.222+0000 7ff3f9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.222+0000 7ff3f9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.627 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.222+0000 7ff3f9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.222+0000 7ff3f9400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.224+0000 7ff3f5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.224+0000 7ff3f5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.224+0000 7ff3f5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.224+0000 7ff3f5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.628 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:46:33.224+0000 7ff3f5800700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:46:33.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.907+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f8f340983d0 con 0x7f8f48102fb0 2024-04-25T18:46:33.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.911+0000 7f8f44c00640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 0x7f8f2c0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:33.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.912+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f8f2c0ad3d0 con 0x7f8f2c0aaaa0 2024-04-25T18:46:33.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.916+0000 7f8f47e00640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 0x7f8f2c0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:33.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:33.936+0000 7f8f47e00640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 0x7f8f2c0ace90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8f3802a930 tx=0x7f8f380058b0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:34.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T18:46:34.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:46:34.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: Active manager daemon x restarted 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: Activating manager daemon x 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: mgrmap e21: x(active, starting, since 0.0133208s), standbys: y, z 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:46:34.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:46:34.069 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:46:34.070 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: Manager daemon x is now available 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:46:34.071 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:33 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:46:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.073+0000 7f8f44c00640 1 -- 172.21.15.63:0/3754348404 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8f2c0ad3d0 con 0x7f8f2c0aaaa0 2024-04-25T18:46:34.074 INFO:teuthology.orchestra.run.smithi063.stdout:Schedule set for path / 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 msgr2=0x7f8f2c0ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 0x7f8f2c0ace90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8f3802a930 tx=0x7f8f380058b0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 msgr2=0x7f8f4819eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 0x7f8f4819eb10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8f34005fd0 tx=0x7f8f3400ee50 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 shutdown_connections 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f2c0aaaa0 0x7f8f2c0ace90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f48102fb0 0x7f8f4819eb10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f48105960 0x7f8f481a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 --2- 172.21.15.63:0/3754348404 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f48100640 0x7f8f4819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.075+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 >> 172.21.15.63:0/3754348404 conn(0x7f8f480fc240 msgr2=0x7f8f48103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.076+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 shutdown_connections 2024-04-25T18:46:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.076+0000 7f8f4d600640 1 -- 172.21.15.63:0/3754348404 wait complete. 2024-04-25T18:46:34.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T18:46:34.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:46:34.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: Active manager daemon x restarted 2024-04-25T18:46:34.091 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: Activating manager daemon x 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: mgrmap e21: x(active, starting, since 0.0133208s), standbys: y, z 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:46:34.092 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:46:34.093 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:46:34.094 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: Manager daemon x is now available 2024-04-25T18:46:34.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:34.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:46:34.095 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:33 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:46:34.096 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4115947392' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: Active manager daemon x restarted 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: Activating manager daemon x 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: mgrmap e21: x(active, starting, since 0.0133208s), standbys: y, z 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T18:46:34.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T18:46:34.205 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: Manager daemon x is now available 2024-04-25T18:46:34.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:34.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T18:46:34.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:33 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T18:46:34.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 -- 172.21.15.63:0/403509428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c105360 msgr2=0x7f9b3c107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 --2- 172.21.15.63:0/403509428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c105360 0x7f9b3c107750 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9b28009900 tx=0x7f9b2802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 -- 172.21.15.63:0/403509428 shutdown_connections 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 --2- 172.21.15.63:0/403509428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c107c90 0x7f9b3c10a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 --2- 172.21.15.63:0/403509428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c105360 0x7f9b3c107750 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 --2- 172.21.15.63:0/403509428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c102a30 0x7f9b3c104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 -- 172.21.15.63:0/403509428 >> 172.21.15.63:0/403509428 conn(0x7f9b3c0fc420 msgr2=0x7f9b3c0fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 -- 172.21.15.63:0/403509428 shutdown_connections 2024-04-25T18:46:34.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.323+0000 7f9b44600640 1 -- 172.21.15.63:0/403509428 wait complete. 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.324+0000 7f9b44600640 1 Processor -- start 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.324+0000 7f9b44600640 1 -- start start 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.324+0000 7f9b44600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c102a30 0x7f9b3c1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b44600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c105360 0x7f9b3c1009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b44600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b44600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c10db40 con 0x7f9b3c102a30 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b44600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c10d9c0 con 0x7f9b3c105360 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b44600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c10dcc0 con 0x7f9b3c107c90 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b41200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c105360 0x7f9b3c1009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50072/0 (socket says 172.21.15.63:50072) 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 -- 172.21.15.63:0/3333379232 learned_addr learned my addr 172.21.15.63:0/3333379232 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b41c00640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c102a30 0x7f9b3c1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 -- 172.21.15.63:0/3333379232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c105360 msgr2=0x7f9b3c1009e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c105360 0x7f9b3c1009e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 -- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c102a30 msgr2=0x7f9b3c1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c102a30 0x7f9b3c1023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.325+0000 7f9b42600640 1 -- 172.21.15.63:0/3333379232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c1017b0 con 0x7f9b3c107c90 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b42600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400eb10 tx=0x7f9b3400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34004b20 con 0x7f9b3c107c90 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c1a22b0 con 0x7f9b3c107c90 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b34004cc0 con 0x7f9b3c107c90 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34010670 con 0x7f9b3c107c90 2024-04-25T18:46:34.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.326+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c1a2780 con 0x7f9b3c107c90 2024-04-25T18:46:34.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.327+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b3c04fce0 con 0x7f9b3c107c90 2024-04-25T18:46:34.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.329+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f9b34005020 con 0x7f9b3c107c90 2024-04-25T18:46:34.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.330+0000 7f9b3a000640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 0x7f9b240a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.330+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9b340d0760 con 0x7f9b3c107c90 2024-04-25T18:46:34.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.331+0000 7f9b41c00640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 0x7f9b240a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.331+0000 7f9b41c00640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 0x7f9b240a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009bd0 tx=0x7f9b30005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:34.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.334+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b3409d2c0 con 0x7f9b3c107c90 2024-04-25T18:46:34.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.584+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f9b3c062430 con 0x7f9b240a3a60 2024-04-25T18:46:34.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.724+0000 7f9b3a000640 1 -- 172.21.15.63:0/3333379232 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f9b3c062430 con 0x7f9b240a3a60 2024-04-25T18:46:34.725 INFO:teuthology.orchestra.run.smithi063.stdout:Retention added to path / 2024-04-25T18:46:34.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 msgr2=0x7f9b240a5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 0x7f9b240a5f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9b30009bd0 tx=0x7f9b30005f90 comp rx=0 tx=0).stop 2024-04-25T18:46:34.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 msgr2=0x7f9b3c100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400eb10 tx=0x7f9b3400c2d0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 shutdown_connections 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9b240a3a60 0x7f9b240a5f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b3c105360 0x7f9b3c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9b3c107c90 0x7f9b3c100f20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 --2- 172.21.15.63:0/3333379232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b3c102a30 0x7f9b3c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 >> 172.21.15.63:0/3333379232 conn(0x7f9b3c0fc420 msgr2=0x7f9b3c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 shutdown_connections 2024-04-25T18:46:34.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.727+0000 7f9b44600640 1 -- 172.21.15.63:0/3333379232 wait complete. 2024-04-25T18:46:34.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.937+0000 7f7a1f400640 1 -- 172.21.15.63:0/3409970346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18101b20 msgr2=0x7f7a18101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.937+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3409970346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18101b20 0x7f7a18101fa0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009ca0 tx=0x7f7a1002a8c0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 -- 172.21.15.63:0/3409970346 shutdown_connections 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3409970346 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18101b20 0x7f7a18101fa0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3409970346 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101180 0x7f7a181015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3409970346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18100820 0x7f7a18100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 -- 172.21.15.63:0/3409970346 >> 172.21.15.63:0/3409970346 conn(0x7f7a180fc420 msgr2=0x7f7a180fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:34.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 -- 172.21.15.63:0/3409970346 shutdown_connections 2024-04-25T18:46:34.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.938+0000 7f7a1f400640 1 -- 172.21.15.63:0/3409970346 wait complete. 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 Processor -- start 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 -- start start 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18101180 0x7f7a1819a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101b20 0x7f7a181a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810fd50 con 0x7f7a18101180 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810fbd0 con 0x7f7a18100820 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1f400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1810fed0 con 0x7f7a18101b20 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55648/0 (socket says 172.21.15.63:55648) 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101b20 0x7f7a181a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.939+0000 7f7a1ca00640 1 -- 172.21.15.63:0/3646476565 learned_addr learned my addr 172.21.15.63:0/3646476565 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a17e00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18101180 0x7f7a1819a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 -- 172.21.15.63:0/3646476565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101b20 msgr2=0x7f7a181a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101b20 0x7f7a181a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 -- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18101180 msgr2=0x7f7a1819a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18101180 0x7f7a1819a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 -- 172.21.15.63:0/3646476565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a181a1f90 con 0x7f7a18100820 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.940+0000 7f7a1ca00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d950 tx=0x7f7a0c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.941+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c017070 con 0x7f7a18100820 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.941+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a0c009c30 con 0x7f7a18100820 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.941+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a181a2280 con 0x7f7a18100820 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.941+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c00f620 con 0x7f7a18100820 2024-04-25T18:46:34.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.941+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a181a2820 con 0x7f7a18100820 2024-04-25T18:46:34.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.942+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a1810b570 con 0x7f7a18100820 2024-04-25T18:46:34.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.944+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7f7a0c010460 con 0x7f7a18100820 2024-04-25T18:46:34.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.946+0000 7f7a15600640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 0x7f7a000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:34.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.946+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7a0c0d0370 con 0x7f7a18100820 2024-04-25T18:46:34.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.946+0000 7f7a17e00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 0x7f7a000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:34.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.947+0000 7f7a17e00640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 0x7f7a000a6300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7a1819b880 tx=0x7f7a04005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:34.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:34.951+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a0c009de0 con 0x7f7a18100820 2024-04-25T18:46:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: [25/Apr/2024:18:46:33] ENGINE Bus STARTING 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: [25/Apr/2024:18:46:33] ENGINE Serving on https://172.21.15.63:7150 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: [25/Apr/2024:18:46:33] ENGINE Serving on http://172.21.15.63:8765 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: [25/Apr/2024:18:46:33] ENGINE Bus STARTED 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: mgrmap e22: x(active, since 1.03815s), standbys: y, z 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:35.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:34 smithi063 ceph-mon[28006]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:35.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.193+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7a18062390 con 0x7f7a000a3e40 2024-04-25T18:46:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: [25/Apr/2024:18:46:33] ENGINE Bus STARTING 2024-04-25T18:46:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: [25/Apr/2024:18:46:33] ENGINE Serving on https://172.21.15.63:7150 2024-04-25T18:46:35.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: [25/Apr/2024:18:46:33] ENGINE Serving on http://172.21.15.63:8765 2024-04-25T18:46:35.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: [25/Apr/2024:18:46:33] ENGINE Bus STARTED 2024-04-25T18:46:35.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: mgrmap e22: x(active, since 1.03815s), standbys: y, z 2024-04-25T18:46:35.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:35.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:34 smithi070 ceph-mon[32820]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:35.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.213+0000 7f7a15600640 1 -- 172.21.15.63:0/3646476565 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f7a18062390 con 0x7f7a000a3e40 2024-04-25T18:46:35.214 INFO:teuthology.orchestra.run.smithi063.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T18:46:33", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-25T18:46:35.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 msgr2=0x7f7a000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 0x7f7a000a6300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7a1819b880 tx=0x7f7a04005f90 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 msgr2=0x7f7a1819a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d950 tx=0x7f7a0c00de10 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 shutdown_connections 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7a000a3e40 0x7f7a000a6300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7a18101b20 0x7f7a181a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a18101180 0x7f7a1819a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 --2- 172.21.15.63:0/3646476565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a18100820 0x7f7a1819a370 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 >> 172.21.15.63:0/3646476565 conn(0x7f7a180fc420 msgr2=0x7f7a180fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 shutdown_connections 2024-04-25T18:46:35.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.217+0000 7f7a1f400640 1 -- 172.21.15.63:0/3646476565 wait complete. 2024-04-25T18:46:35.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T18:46:35.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: [25/Apr/2024:18:46:33] ENGINE Bus STARTING 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: [25/Apr/2024:18:46:33] ENGINE Serving on https://172.21.15.63:7150 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: [25/Apr/2024:18:46:33] ENGINE Serving on http://172.21.15.63:8765 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: [25/Apr/2024:18:46:33] ENGINE Bus STARTED 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: mgrmap e22: x(active, since 1.03815s), standbys: y, z 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: from='client.25103 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:35.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:34 smithi114 ceph-mon[32988]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:35.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 -- 172.21.15.63:0/3654716624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681028b0 msgr2=0x7fa568104ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 --2- 172.21.15.63:0/3654716624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681028b0 0x7fa568104ca0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa558009900 tx=0x7fa55802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 -- 172.21.15.63:0/3654716624 shutdown_connections 2024-04-25T18:46:35.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 --2- 172.21.15.63:0/3654716624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa568109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 --2- 172.21.15.63:0/3654716624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681051e0 0x7fa5681075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 --2- 172.21.15.63:0/3654716624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681028b0 0x7fa568104ca0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 -- 172.21.15.63:0/3654716624 >> 172.21.15.63:0/3654716624 conn(0x7fa5680fc240 msgr2=0x7fa5680fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:35.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 -- 172.21.15.63:0/3654716624 shutdown_connections 2024-04-25T18:46:35.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.458+0000 7fa56ca00640 1 -- 172.21.15.63:0/3654716624 wait complete. 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.459+0000 7fa56ca00640 1 Processor -- start 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.459+0000 7fa56ca00640 1 -- start start 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.459+0000 7fa56ca00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681028b0 0x7fa56819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.459+0000 7fa56ca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 0x7fa56819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa56ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa56ca00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810d940 con 0x7fa5681028b0 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa56ca00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810d7c0 con 0x7fa568107b10 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa56ca00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa56810dac0 con 0x7fa5681051e0 2024-04-25T18:46:35.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa565e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 0x7fa56819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa566800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681028b0 0x7fa56819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55672/0 (socket says 172.21.15.63:55672) 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa565e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 0x7fa56819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50118/0 (socket says 172.21.15.63:50118) 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 -- 172.21.15.63:0/1791147970 learned_addr learned my addr 172.21.15.63:0/1791147970 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 -- 172.21.15.63:0/1791147970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 msgr2=0x7fa56819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 0x7fa56819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 -- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681028b0 msgr2=0x7fa56819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681028b0 0x7fa56819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.460+0000 7fa567200640 1 -- 172.21.15.63:0/1791147970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5681a1e50 con 0x7fa568107b10 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa567200640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa56000efd0 tx=0x7fa56000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560003b70 con 0x7fa568107b10 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa560003d10 con 0x7fa568107b10 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5681a2140 con 0x7fa568107b10 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600108a0 con 0x7fa568107b10 2024-04-25T18:46:35.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.461+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5681a2690 con 0x7fa568107b10 2024-04-25T18:46:35.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.462+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa568109600 con 0x7fa568107b10 2024-04-25T18:46:35.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.464+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 23) ==== 138766+0+0 (secure 0 0 0) 0x7fa560004020 con 0x7fa568107b10 2024-04-25T18:46:35.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.466+0000 7fa55f400640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 0x7fa54c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:35.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.466+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa5600d0bf0 con 0x7fa568107b10 2024-04-25T18:46:35.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.466+0000 7fa566800640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 0x7fa54c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:35.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.467+0000 7fa566800640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 0x7fa54c0a6250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa558031500 tx=0x7fa558005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:35.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.470+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa56009d220 con 0x7fa568107b10 2024-04-25T18:46:35.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.717+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa568062390 con 0x7fa54c0a3d90 2024-04-25T18:46:35.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.726+0000 7fa55f400640 1 -- 172.21.15.63:0/1791147970 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa568062390 con 0x7fa54c0a3d90 2024-04-25T18:46:35.727 INFO:teuthology.orchestra.run.smithi063.stdout:/ 1m 6m3h 2024-04-25T18:46:35.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 msgr2=0x7fa54c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 0x7fa54c0a6250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa558031500 tx=0x7fa558005a20 comp rx=0 tx=0).stop 2024-04-25T18:46:35.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 msgr2=0x7fa5681a17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:35.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa56000efd0 tx=0x7fa56000c6a0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 shutdown_connections 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa54c0a3d90 0x7fa54c0a6250 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa568107b10 0x7fa5681a17e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5681051e0 0x7fa56819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 --2- 172.21.15.63:0/1791147970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5681028b0 0x7fa56819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 >> 172.21.15.63:0/1791147970 conn(0x7fa5680fc240 msgr2=0x7fa568101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 shutdown_connections 2024-04-25T18:46:35.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:35.728+0000 7fa56ca00640 1 -- 172.21.15.63:0/1791147970 wait complete. 2024-04-25T18:46:35.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-25T18:46:35.826 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-25T18:46:35.836 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T18:46:35.852 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T18:46:35.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T18:46:35.857 INFO:teuthology.run_tasks:Running task quiescer... 2024-04-25T18:46:35.869 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T18:46:35.869 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:46:36.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='client.25138 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:36.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T18:46:36.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:35 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='client.25138 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:35 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.314 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='client.25138 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: mgrmap e23: x(active, since 2s), standbys: y, z 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:36.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:35 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:46:37.566 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:37 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:46:37.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:37 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T18:46:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:37.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:46:37.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:37 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:46:38.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 -- 172.21.15.63:0/3372278291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a2806a750 msgr2=0x7f1a2806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:38.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/3372278291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a2806a750 0x7f1a2806abb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009990 tx=0x7f1a14009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 -- 172.21.15.63:0/3372278291 shutdown_connections 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/3372278291 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a28108a30 0x7f1a28108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/3372278291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/3372278291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a2806a750 0x7f1a2806abb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 -- 172.21.15.63:0/3372278291 >> 172.21.15.63:0/3372278291 conn(0x7f1a280788f0 msgr2=0x7f1a28078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 -- 172.21.15.63:0/3372278291 shutdown_connections 2024-04-25T18:46:38.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.504+0000 7f1a2ea00700 1 -- 172.21.15.63:0/3372278291 wait complete. 2024-04-25T18:46:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.505+0000 7f1a2ea00700 1 Processor -- start 2024-04-25T18:46:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.505+0000 7f1a2ea00700 1 -- start start 2024-04-25T18:46:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 0x7f1a2819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:38.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28112c20 con 0x7f1a2806b0f0 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28112aa0 con 0x7f1a28108a30 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ea00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1a28112da0 con 0x7f1a2806a750 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a27400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a27e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 0x7f1a2819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a27400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57756/0 (socket says 172.21.15.63:57756) 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a2ca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43320/0 (socket says 172.21.15.63:43320) 2024-04-25T18:46:38.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.506+0000 7f1a27400700 1 -- 172.21.15.63:0/672449641 learned_addr learned my addr 172.21.15.63:0/672449641 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 -- 172.21.15.63:0/672449641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 msgr2=0x7f1a2819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 -- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 msgr2=0x7f1a2819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 -- 172.21.15.63:0/672449641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a281a03b0 con 0x7f1a2806a750 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27400700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a2ca00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.507+0000 7f1a27e00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 0x7f1a2819b3a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e9f0 tx=0x7f1a1800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:38.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.508+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18004220 con 0x7f1a2806a750 2024-04-25T18:46:38.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.508+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a281a06a0 con 0x7f1a2806a750 2024-04-25T18:46:38.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.508+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a180043c0 con 0x7f1a2806a750 2024-04-25T18:46:38.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.508+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800ce60 con 0x7f1a2806a750 2024-04-25T18:46:38.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.508+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a281a7f20 con 0x7f1a2806a750 2024-04-25T18:46:38.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.509+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a28069730 con 0x7f1a2806a750 2024-04-25T18:46:38.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.512+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f1a180052f0 con 0x7f1a2806a750 2024-04-25T18:46:38.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.513+0000 7f1a24c00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 0x7f1a0c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:38.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.513+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f1a180d0a40 con 0x7f1a2806a750 2024-04-25T18:46:38.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.514+0000 7f1a27400700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 0x7f1a0c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:38.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.514+0000 7f1a27400700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 0x7f1a0c0a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a28078f40 tx=0x7f1a1401a040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:38.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.518+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a1809cf00 con 0x7f1a2806a750 2024-04-25T18:46:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:38 smithi063 ceph-mon[28006]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:38 smithi063 ceph-mon[28006]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T18:46:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:38 smithi070 ceph-mon[32820]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:38 smithi070 ceph-mon[32820]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T18:46:38.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.757+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a28062e80 con 0x7f1a0c0a3ec0 2024-04-25T18:46:38.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.767+0000 7f1a24c00700 1 -- 172.21.15.63:0/672449641 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257796 (secure 0 0 0) 0x7f1a28062e80 con 0x7f1a0c0a3ec0 2024-04-25T18:46:38.774 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.772+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 msgr2=0x7f1a0c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.772+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 0x7f1a0c0a6380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1a28078f40 tx=0x7f1a1401a040 comp rx=0 tx=0).stop 2024-04-25T18:46:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.772+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 msgr2=0x7f1a2819b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.772+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 0x7f1a2819b3a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800e9f0 tx=0x7f1a1800ed00 comp rx=0 tx=0).stop 2024-04-25T18:46:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 shutdown_connections 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1a0c0a3ec0 0x7f1a0c0a6380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1a2806a750 0x7f1a2819b3a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a28108a30 0x7f1a2819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 --2- 172.21.15.63:0/672449641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a2806b0f0 0x7f1a2819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 >> 172.21.15.63:0/672449641 conn(0x7f1a280788f0 msgr2=0x7f1a2810dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 shutdown_connections 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:38.773+0000 7f1a2ea00700 1 -- 172.21.15.63:0/672449641 wait complete. 2024-04-25T18:46:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:46:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:38 smithi114 ceph-mon[32988]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:38 smithi114 ceph-mon[32988]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-04-25T18:46:39.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:39.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:39.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.855 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:39.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:39.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:39.857 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:39 smithi063 ceph-mon[28006]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:39.858 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-25T18:46:36.910247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1187910,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1576,"num_read_kb":939,"num_write":477,"num_write_kb":4379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":398064,"kb_used_data":15536,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124543760,"statfs":{"total":1151940427776,"available":1151532810240,"internally_reserved":0,"allocated":15908864,"data_stored":9148217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.961917"},"pg_stats":[{"pgid":"3.3f","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422858+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:46:33.422858+0000","last_peered":"2024-04-25T18:46:33.422858+0000","last_clean":"2024-04-25T18:46:33.422858+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:46:33.422858+0000","last_undegraded":"2024-04-25T18:46:33.422858+0000","last_fullsized":"2024-04-25T18:46:33.422858+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424681+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:46:33.424681+0000","last_peered":"2024-04-25T18:46:33.424681+0000","last_clean":"2024-04-25T18:46:33.424681+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:46:33.424681+0000","last_undegraded":"2024-04-25T18:46:33.424681+0000","last_fullsized":"2024-04-25T18:46:33.424681+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424189+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:46:33.424189+0000","last_peered":"2024-04-25T18:46:33.424189+0000","last_clean":"2024-04-25T18:46:33.424189+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:46:33.424189+0000","last_undegraded":"2024-04-25T18:46:33.424189+0000","last_fullsized":"2024-04-25T18:46:33.424189+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890407+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:46:32.890407+0000","last_peered":"2024-04-25T18:46:32.890407+0000","last_clean":"2024-04-25T18:46:32.890407+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:46:32.890407+0000","last_undegraded":"2024-04-25T18:46:32.890407+0000","last_fullsized":"2024-04-25T18:46:32.890407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023708+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:46:33.023708+0000","last_peered":"2024-04-25T18:46:33.023708+0000","last_clean":"2024-04-25T18:46:33.023708+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:46:33.023708+0000","last_undegraded":"2024-04-25T18:46:33.023708+0000","last_fullsized":"2024-04-25T18:46:33.023708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933634+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:46:32.933634+0000","last_peered":"2024-04-25T18:46:32.933634+0000","last_clean":"2024-04-25T18:46:32.933634+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:46:32.933634+0000","last_undegraded":"2024-04-25T18:46:32.933634+0000","last_fullsized":"2024-04-25T18:46:32.933634+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231868+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:46:33.231868+0000","last_peered":"2024-04-25T18:46:33.231868+0000","last_clean":"2024-04-25T18:46:33.231868+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:46:33.231868+0000","last_undegraded":"2024-04-25T18:46:33.231868+0000","last_fullsized":"2024-04-25T18:46:33.231868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023428+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:46:33.023428+0000","last_peered":"2024-04-25T18:46:33.023428+0000","last_clean":"2024-04-25T18:46:33.023428+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:46:33.023428+0000","last_undegraded":"2024-04-25T18:46:33.023428+0000","last_fullsized":"2024-04-25T18:46:33.023428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238908+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:46:33.238908+0000","last_peered":"2024-04-25T18:46:33.238908+0000","last_clean":"2024-04-25T18:46:33.238908+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:46:33.238908+0000","last_undegraded":"2024-04-25T18:46:33.238908+0000","last_fullsized":"2024-04-25T18:46:33.238908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"96'16","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933875+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:46:32.933875+0000","last_peered":"2024-04-25T18:46:32.933875+0000","last_clean":"2024-04-25T18:46:32.933875+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:46:32.933875+0000","last_undegraded":"2024-04-25T18:46:32.933875+0000","last_fullsized":"2024-04-25T18:46:32.933875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232039+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:46:33.232039+0000","last_peered":"2024-04-25T18:46:33.232039+0000","last_clean":"2024-04-25T18:46:33.232039+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:46:33.232039+0000","last_undegraded":"2024-04-25T18:46:33.232039+0000","last_fullsized":"2024-04-25T18:46:33.232039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890594+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:46:32.890594+0000","last_peered":"2024-04-25T18:46:32.890594+0000","last_clean":"2024-04-25T18:46:32.890594+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:46:32.890594+0000","last_undegraded":"2024-04-25T18:46:32.890594+0000","last_fullsized":"2024-04-25T18:46:32.890594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095489+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:46:33.095489+0000","last_peered":"2024-04-25T18:46:33.095489+0000","last_clean":"2024-04-25T18:46:33.095489+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:46:33.095489+0000","last_undegraded":"2024-04-25T18:46:33.095489+0000","last_fullsized":"2024-04-25T18:46:33.095489+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423834+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:46:33.423834+0000","last_peered":"2024-04-25T18:46:33.423834+0000","last_clean":"2024-04-25T18:46:33.423834+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:46:33.423834+0000","last_undegraded":"2024-04-25T18:46:33.423834+0000","last_fullsized":"2024-04-25T18:46:33.423834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423857+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:46:33.423857+0000","last_peered":"2024-04-25T18:46:33.423857+0000","last_clean":"2024-04-25T18:46:33.423857+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:46:33.423857+0000","last_undegraded":"2024-04-25T18:46:33.423857+0000","last_fullsized":"2024-04-25T18:46:33.423857+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238084+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:46:33.238084+0000","last_peered":"2024-04-25T18:46:33.238084+0000","last_clean":"2024-04-25T18:46:33.238084+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:46:33.238084+0000","last_undegraded":"2024-04-25T18:46:33.238084+0000","last_fullsized":"2024-04-25T18:46:33.238084+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422975+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:46:33.422975+0000","last_peered":"2024-04-25T18:46:33.422975+0000","last_clean":"2024-04-25T18:46:33.422975+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:46:33.422975+0000","last_undegraded":"2024-04-25T18:46:33.422975+0000","last_fullsized":"2024-04-25T18:46:33.422975+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889201+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:46:32.889201+0000","last_peered":"2024-04-25T18:46:32.889201+0000","last_clean":"2024-04-25T18:46:32.889201+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:46:32.889201+0000","last_undegraded":"2024-04-25T18:46:32.889201+0000","last_fullsized":"2024-04-25T18:46:32.889201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892794+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:46:32.892794+0000","last_peered":"2024-04-25T18:46:32.892794+0000","last_clean":"2024-04-25T18:46:32.892794+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:46:32.892794+0000","last_undegraded":"2024-04-25T18:46:32.892794+0000","last_fullsized":"2024-04-25T18:46:32.892794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889557+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:46:32.889557+0000","last_peered":"2024-04-25T18:46:32.889557+0000","last_clean":"2024-04-25T18:46:32.889557+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:46:32.889557+0000","last_undegraded":"2024-04-25T18:46:32.889557+0000","last_fullsized":"2024-04-25T18:46:32.889557+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024156+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:46:33.024156+0000","last_peered":"2024-04-25T18:46:33.024156+0000","last_clean":"2024-04-25T18:46:33.024156+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:46:33.024156+0000","last_undegraded":"2024-04-25T18:46:33.024156+0000","last_fullsized":"2024-04-25T18:46:33.024156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024791+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:46:33.024791+0000","last_peered":"2024-04-25T18:46:33.024791+0000","last_clean":"2024-04-25T18:46:33.024791+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:46:33.024791+0000","last_undegraded":"2024-04-25T18:46:33.024791+0000","last_fullsized":"2024-04-25T18:46:33.024791+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424904+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:46:33.424904+0000","last_peered":"2024-04-25T18:46:33.424904+0000","last_clean":"2024-04-25T18:46:33.424904+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:46:33.424904+0000","last_undegraded":"2024-04-25T18:46:33.424904+0000","last_fullsized":"2024-04-25T18:46:33.424904+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024166+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:46:33.024166+0000","last_peered":"2024-04-25T18:46:33.024166+0000","last_clean":"2024-04-25T18:46:33.024166+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:46:33.024166+0000","last_undegraded":"2024-04-25T18:46:33.024166+0000","last_fullsized":"2024-04-25T18:46:33.024166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238451+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:46:33.238451+0000","last_peered":"2024-04-25T18:46:33.238451+0000","last_clean":"2024-04-25T18:46:33.238451+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:46:33.238451+0000","last_undegraded":"2024-04-25T18:46:33.238451+0000","last_fullsized":"2024-04-25T18:46:33.238451+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024005+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:46:33.024005+0000","last_peered":"2024-04-25T18:46:33.024005+0000","last_clean":"2024-04-25T18:46:33.024005+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:46:33.024005+0000","last_undegraded":"2024-04-25T18:46:33.024005+0000","last_fullsized":"2024-04-25T18:46:33.024005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"96'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936393+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:46:32.936393+0000","last_peered":"2024-04-25T18:46:32.936393+0000","last_clean":"2024-04-25T18:46:32.936393+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:46:32.936393+0000","last_undegraded":"2024-04-25T18:46:32.936393+0000","last_fullsized":"2024-04-25T18:46:32.936393+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"97'7","reported_seq":262,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231591+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:46:33.231591+0000","last_peered":"2024-04-25T18:46:33.231591+0000","last_clean":"2024-04-25T18:46:33.231591+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:46:33.231591+0000","last_undegraded":"2024-04-25T18:46:33.231591+0000","last_fullsized":"2024-04-25T18:46:33.231591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":222,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"96'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023638+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:46:33.023638+0000","last_peered":"2024-04-25T18:46:33.023638+0000","last_clean":"2024-04-25T18:46:33.023638+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:46:33.023638+0000","last_undegraded":"2024-04-25T18:46:33.023638+0000","last_fullsized":"2024-04-25T18:46:33.023638+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933556+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:46:32.933556+0000","last_peered":"2024-04-25T18:46:32.933556+0000","last_clean":"2024-04-25T18:46:32.933556+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:46:32.933556+0000","last_undegraded":"2024-04-25T18:46:32.933556+0000","last_fullsized":"2024-04-25T18:46:32.933556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936363+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:46:32.936363+0000","last_peered":"2024-04-25T18:46:32.936363+0000","last_clean":"2024-04-25T18:46:32.936363+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:46:32.936363+0000","last_undegraded":"2024-04-25T18:46:32.936363+0000","last_fullsized":"2024-04-25T18:46:32.936363+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095036+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:46:33.095036+0000","last_peered":"2024-04-25T18:46:33.095036+0000","last_clean":"2024-04-25T18:46:33.095036+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:46:33.095036+0000","last_undegraded":"2024-04-25T18:46:33.095036+0000","last_fullsized":"2024-04-25T18:46:33.095036+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023495+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:46:33.023495+0000","last_peered":"2024-04-25T18:46:33.023495+0000","last_clean":"2024-04-25T18:46:33.023495+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:46:33.023495+0000","last_undegraded":"2024-04-25T18:46:33.023495+0000","last_fullsized":"2024-04-25T18:46:33.023495+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936092+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:46:32.936092+0000","last_peered":"2024-04-25T18:46:32.936092+0000","last_clean":"2024-04-25T18:46:32.936092+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:46:32.936092+0000","last_undegraded":"2024-04-25T18:46:32.936092+0000","last_fullsized":"2024-04-25T18:46:32.936092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889094+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:46:32.889094+0000","last_peered":"2024-04-25T18:46:32.889094+0000","last_clean":"2024-04-25T18:46:32.889094+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:46:32.889094+0000","last_undegraded":"2024-04-25T18:46:32.889094+0000","last_fullsized":"2024-04-25T18:46:32.889094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":257,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.934112+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:46:32.934112+0000","last_peered":"2024-04-25T18:46:32.934112+0000","last_clean":"2024-04-25T18:46:32.934112+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:46:32.934112+0000","last_undegraded":"2024-04-25T18:46:32.934112+0000","last_fullsized":"2024-04-25T18:46:32.934112+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238717+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:46:33.238717+0000","last_peered":"2024-04-25T18:46:33.238717+0000","last_clean":"2024-04-25T18:46:33.238717+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:46:33.238717+0000","last_undegraded":"2024-04-25T18:46:33.238717+0000","last_fullsized":"2024-04-25T18:46:33.238717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422655+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:46:33.422655+0000","last_peered":"2024-04-25T18:46:33.422655+0000","last_clean":"2024-04-25T18:46:33.422655+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:46:33.422655+0000","last_undegraded":"2024-04-25T18:46:33.422655+0000","last_fullsized":"2024-04-25T18:46:33.422655+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.425055+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:46:33.425055+0000","last_peered":"2024-04-25T18:46:33.425055+0000","last_clean":"2024-04-25T18:46:33.425055+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:46:33.425055+0000","last_undegraded":"2024-04-25T18:46:33.425055+0000","last_fullsized":"2024-04-25T18:46:33.425055+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936480+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:46:32.936480+0000","last_peered":"2024-04-25T18:46:32.936480+0000","last_clean":"2024-04-25T18:46:32.936480+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:46:32.936480+0000","last_undegraded":"2024-04-25T18:46:32.936480+0000","last_fullsized":"2024-04-25T18:46:32.936480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"96'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238458+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:46:33.238458+0000","last_peered":"2024-04-25T18:46:33.238458+0000","last_clean":"2024-04-25T18:46:33.238458+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:46:33.238458+0000","last_undegraded":"2024-04-25T18:46:33.238458+0000","last_fullsized":"2024-04-25T18:46:33.238458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"97'36","reported_seq":520,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.237914+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:46:33.237914+0000","last_peered":"2024-04-25T18:46:33.237914+0000","last_clean":"2024-04-25T18:46:33.237914+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:46:33.237914+0000","last_undegraded":"2024-04-25T18:46:33.237914+0000","last_fullsized":"2024-04-25T18:46:33.237914+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66476,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":451,"num_read_kb":284,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023970+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:46:33.023970+0000","last_peered":"2024-04-25T18:46:33.023970+0000","last_clean":"2024-04-25T18:46:33.023970+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:46:33.023970+0000","last_undegraded":"2024-04-25T18:46:33.023970+0000","last_fullsized":"2024-04-25T18:46:33.023970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892434+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:46:32.892434+0000","last_peered":"2024-04-25T18:46:32.892434+0000","last_clean":"2024-04-25T18:46:32.892434+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:46:32.892434+0000","last_undegraded":"2024-04-25T18:46:32.892434+0000","last_fullsized":"2024-04-25T18:46:32.892434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094919+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:46:33.094919+0000","last_peered":"2024-04-25T18:46:33.094919+0000","last_clean":"2024-04-25T18:46:33.094919+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:46:33.094919+0000","last_undegraded":"2024-04-25T18:46:33.094919+0000","last_fullsized":"2024-04-25T18:46:33.094919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936173+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:46:32.936173+0000","last_peered":"2024-04-25T18:46:32.936173+0000","last_clean":"2024-04-25T18:46:32.936173+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:46:32.936173+0000","last_undegraded":"2024-04-25T18:46:32.936173+0000","last_fullsized":"2024-04-25T18:46:32.936173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232141+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:46:33.232141+0000","last_peered":"2024-04-25T18:46:33.232141+0000","last_clean":"2024-04-25T18:46:33.232141+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:46:33.232141+0000","last_undegraded":"2024-04-25T18:46:33.232141+0000","last_fullsized":"2024-04-25T18:46:33.232141+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"97'8","reported_seq":266,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.234856+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:46:33.234856+0000","last_peered":"2024-04-25T18:46:33.234856+0000","last_clean":"2024-04-25T18:46:33.234856+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:46:33.234856+0000","last_undegraded":"2024-04-25T18:46:33.234856+0000","last_fullsized":"2024-04-25T18:46:33.234856+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":226,"num_read_kb":6,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231813+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:46:33.231813+0000","last_peered":"2024-04-25T18:46:33.231813+0000","last_clean":"2024-04-25T18:46:33.231813+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:46:33.231813+0000","last_undegraded":"2024-04-25T18:46:33.231813+0000","last_fullsized":"2024-04-25T18:46:33.231813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422514+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:46:33.422514+0000","last_peered":"2024-04-25T18:46:33.422514+0000","last_clean":"2024-04-25T18:46:33.422514+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:46:33.422514+0000","last_undegraded":"2024-04-25T18:46:33.422514+0000","last_fullsized":"2024-04-25T18:46:33.422514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936373+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:46:32.936373+0000","last_peered":"2024-04-25T18:46:32.936373+0000","last_clean":"2024-04-25T18:46:32.936373+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:46:32.936373+0000","last_undegraded":"2024-04-25T18:46:32.936373+0000","last_fullsized":"2024-04-25T18:46:32.936373+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024089+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:46:33.024089+0000","last_peered":"2024-04-25T18:46:33.024089+0000","last_clean":"2024-04-25T18:46:33.024089+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:46:33.024089+0000","last_undegraded":"2024-04-25T18:46:33.024089+0000","last_fullsized":"2024-04-25T18:46:33.024089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422673+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:46:33.422673+0000","last_peered":"2024-04-25T18:46:33.422673+0000","last_clean":"2024-04-25T18:46:33.422673+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:46:33.422673+0000","last_undegraded":"2024-04-25T18:46:33.422673+0000","last_fullsized":"2024-04-25T18:46:33.422673+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889070+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:46:32.889070+0000","last_peered":"2024-04-25T18:46:32.889070+0000","last_clean":"2024-04-25T18:46:32.889070+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:46:32.889070+0000","last_undegraded":"2024-04-25T18:46:32.889070+0000","last_fullsized":"2024-04-25T18:46:32.889070+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933754+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:46:32.933754+0000","last_peered":"2024-04-25T18:46:32.933754+0000","last_clean":"2024-04-25T18:46:32.933754+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:46:32.933754+0000","last_undegraded":"2024-04-25T18:46:32.933754+0000","last_fullsized":"2024-04-25T18:46:32.933754+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232202+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:46:33.232202+0000","last_peered":"2024-04-25T18:46:33.232202+0000","last_clean":"2024-04-25T18:46:33.232202+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:46:33.232202+0000","last_undegraded":"2024-04-25T18:46:33.232202+0000","last_fullsized":"2024-04-25T18:46:33.232202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933629+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:46:32.933629+0000","last_peered":"2024-04-25T18:46:32.933629+0000","last_clean":"2024-04-25T18:46:32.933629+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:46:32.933629+0000","last_undegraded":"2024-04-25T18:46:32.933629+0000","last_fullsized":"2024-04-25T18:46:32.933629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.888871+0000","last_change":"2024-04-25T18:43:55.423277+0000","last_active":"2024-04-25T18:46:32.888871+0000","last_peered":"2024-04-25T18:46:32.888871+0000","last_clean":"2024-04-25T18:46:32.888871+0000","last_became_active":"2024-04-25T18:43:55.421960+0000","last_became_peered":"2024-04-25T18:43:55.421960+0000","last_unstale":"2024-04-25T18:46:32.888871+0000","last_undegraded":"2024-04-25T18:46:32.888871+0000","last_fullsized":"2024-04-25T18:46:32.888871+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:27.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":260,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095506+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:46:33.095506+0000","last_peered":"2024-04-25T18:46:33.095506+0000","last_clean":"2024-04-25T18:46:33.095506+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:46:33.095506+0000","last_undegraded":"2024-04-25T18:46:33.095506+0000","last_fullsized":"2024-04-25T18:46:33.095506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":220,"num_read_kb":220,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024020+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:46:33.024020+0000","last_peered":"2024-04-25T18:46:33.024020+0000","last_clean":"2024-04-25T18:46:33.024020+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:46:33.024020+0000","last_undegraded":"2024-04-25T18:46:33.024020+0000","last_fullsized":"2024-04-25T18:46:33.024020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890216+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:46:32.890216+0000","last_peered":"2024-04-25T18:46:32.890216+0000","last_clean":"2024-04-25T18:46:32.890216+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:46:32.890216+0000","last_undegraded":"2024-04-25T18:46:32.890216+0000","last_fullsized":"2024-04-25T18:46:32.890216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":369,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231929+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:46:33.231929+0000","last_peered":"2024-04-25T18:46:33.231929+0000","last_clean":"2024-04-25T18:46:33.231929+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:46:33.231929+0000","last_undegraded":"2024-04-25T18:46:33.231929+0000","last_fullsized":"2024-04-25T18:46:33.231929+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094731+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:46:33.094731+0000","last_peered":"2024-04-25T18:46:33.094731+0000","last_clean":"2024-04-25T18:46:33.094731+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:46:33.094731+0000","last_undegraded":"2024-04-25T18:46:33.094731+0000","last_fullsized":"2024-04-25T18:46:33.094731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232115+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:46:33.232115+0000","last_peered":"2024-04-25T18:46:33.232115+0000","last_clean":"2024-04-25T18:46:33.232115+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:46:33.232115+0000","last_undegraded":"2024-04-25T18:46:33.232115+0000","last_fullsized":"2024-04-25T18:46:33.232115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024041+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:46:33.024041+0000","last_peered":"2024-04-25T18:46:33.024041+0000","last_clean":"2024-04-25T18:46:33.024041+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:46:33.024041+0000","last_undegraded":"2024-04-25T18:46:33.024041+0000","last_fullsized":"2024-04-25T18:46:33.024041+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094993+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:46:33.094993+0000","last_peered":"2024-04-25T18:46:33.094993+0000","last_clean":"2024-04-25T18:46:33.094993+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:46:33.094993+0000","last_undegraded":"2024-04-25T18:46:33.094993+0000","last_fullsized":"2024-04-25T18:46:33.094993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"97'47","reported_seq":103,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.180362+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:46:33.180362+0000","last_peered":"2024-04-25T18:46:33.180362+0000","last_clean":"2024-04-25T18:46:33.180362+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:46:33.180362+0000","last_undegraded":"2024-04-25T18:46:33.180362+0000","last_fullsized":"2024-04-25T18:46:33.180362+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":85,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024773+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:46:33.024773+0000","last_peered":"2024-04-25T18:46:33.024773+0000","last_clean":"2024-04-25T18:46:33.024773+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:46:33.024773+0000","last_undegraded":"2024-04-25T18:46:33.024773+0000","last_fullsized":"2024-04-25T18:46:33.024773+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890406+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:46:32.890406+0000","last_peered":"2024-04-25T18:46:32.890406+0000","last_clean":"2024-04-25T18:46:32.890406+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:46:32.890406+0000","last_undegraded":"2024-04-25T18:46:32.890406+0000","last_fullsized":"2024-04-25T18:46:32.890406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424814+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:46:33.424814+0000","last_peered":"2024-04-25T18:46:33.424814+0000","last_clean":"2024-04-25T18:46:33.424814+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:46:33.424814+0000","last_undegraded":"2024-04-25T18:46:33.424814+0000","last_fullsized":"2024-04-25T18:46:33.424814+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238097+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:46:33.238097+0000","last_peered":"2024-04-25T18:46:33.238097+0000","last_clean":"2024-04-25T18:46:33.238097+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:46:33.238097+0000","last_undegraded":"2024-04-25T18:46:33.238097+0000","last_fullsized":"2024-04-25T18:46:33.238097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023626+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:46:33.023626+0000","last_peered":"2024-04-25T18:46:33.023626+0000","last_clean":"2024-04-25T18:46:33.023626+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:46:33.023626+0000","last_undegraded":"2024-04-25T18:46:33.023626+0000","last_fullsized":"2024-04-25T18:46:33.023626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023651+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:46:33.023651+0000","last_peered":"2024-04-25T18:46:33.023651+0000","last_clean":"2024-04-25T18:46:33.023651+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:46:33.023651+0000","last_undegraded":"2024-04-25T18:46:33.023651+0000","last_fullsized":"2024-04-25T18:46:33.023651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422592+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:46:33.422592+0000","last_peered":"2024-04-25T18:46:33.422592+0000","last_clean":"2024-04-25T18:46:33.422592+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:46:33.422592+0000","last_undegraded":"2024-04-25T18:46:33.422592+0000","last_fullsized":"2024-04-25T18:46:33.422592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892547+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:46:32.892547+0000","last_peered":"2024-04-25T18:46:32.892547+0000","last_clean":"2024-04-25T18:46:32.892547+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:46:32.892547+0000","last_undegraded":"2024-04-25T18:46:32.892547+0000","last_fullsized":"2024-04-25T18:46:32.892547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024105+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:46:33.024105+0000","last_peered":"2024-04-25T18:46:33.024105+0000","last_clean":"2024-04-25T18:46:33.024105+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:46:33.024105+0000","last_undegraded":"2024-04-25T18:46:33.024105+0000","last_fullsized":"2024-04-25T18:46:33.024105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023735+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:46:33.023735+0000","last_peered":"2024-04-25T18:46:33.023735+0000","last_clean":"2024-04-25T18:46:33.023735+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:46:33.023735+0000","last_undegraded":"2024-04-25T18:46:33.023735+0000","last_fullsized":"2024-04-25T18:46:33.023735+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231981+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:46:33.231981+0000","last_peered":"2024-04-25T18:46:33.231981+0000","last_clean":"2024-04-25T18:46:33.231981+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:46:33.231981+0000","last_undegraded":"2024-04-25T18:46:33.231981+0000","last_fullsized":"2024-04-25T18:46:33.231981+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892740+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:46:32.892740+0000","last_peered":"2024-04-25T18:46:32.892740+0000","last_clean":"2024-04-25T18:46:32.892740+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:46:32.892740+0000","last_undegraded":"2024-04-25T18:46:32.892740+0000","last_fullsized":"2024-04-25T18:46:32.892740+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232147+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:46:33.232147+0000","last_peered":"2024-04-25T18:46:33.232147+0000","last_clean":"2024-04-25T18:46:33.232147+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:46:33.232147+0000","last_undegraded":"2024-04-25T18:46:33.232147+0000","last_fullsized":"2024-04-25T18:46:33.232147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"97'39","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.176323+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:46:33.176323+0000","last_peered":"2024-04-25T18:46:33.176323+0000","last_clean":"2024-04-25T18:46:33.176323+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:46:33.176323+0000","last_undegraded":"2024-04-25T18:46:33.176323+0000","last_fullsized":"2024-04-25T18:46:33.176323+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232014+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:46:33.232014+0000","last_peered":"2024-04-25T18:46:33.232014+0000","last_clean":"2024-04-25T18:46:33.232014+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:46:33.232014+0000","last_undegraded":"2024-04-25T18:46:33.232014+0000","last_fullsized":"2024-04-25T18:46:33.232014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094761+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:46:33.094761+0000","last_peered":"2024-04-25T18:46:33.094761+0000","last_clean":"2024-04-25T18:46:33.094761+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:46:33.094761+0000","last_undegraded":"2024-04-25T18:46:33.094761+0000","last_fullsized":"2024-04-25T18:46:33.094761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232252+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:46:33.232252+0000","last_peered":"2024-04-25T18:46:33.232252+0000","last_clean":"2024-04-25T18:46:33.232252+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:46:33.232252+0000","last_undegraded":"2024-04-25T18:46:33.232252+0000","last_fullsized":"2024-04-25T18:46:33.232252+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422717+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:46:33.422717+0000","last_peered":"2024-04-25T18:46:33.422717+0000","last_clean":"2024-04-25T18:46:33.422717+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:46:33.422717+0000","last_undegraded":"2024-04-25T18:46:33.422717+0000","last_fullsized":"2024-04-25T18:46:33.422717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"96'12","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892408+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:46:32.892408+0000","last_peered":"2024-04-25T18:46:32.892408+0000","last_clean":"2024-04-25T18:46:32.892408+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:46:32.892408+0000","last_undegraded":"2024-04-25T18:46:32.892408+0000","last_fullsized":"2024-04-25T18:46:32.892408+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892321+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:46:32.892321+0000","last_peered":"2024-04-25T18:46:32.892321+0000","last_clean":"2024-04-25T18:46:32.892321+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:46:32.892321+0000","last_undegraded":"2024-04-25T18:46:32.892321+0000","last_fullsized":"2024-04-25T18:46:32.892321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424292+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:46:33.424292+0000","last_peered":"2024-04-25T18:46:33.424292+0000","last_clean":"2024-04-25T18:46:33.424292+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:46:33.424292+0000","last_undegraded":"2024-04-25T18:46:33.424292+0000","last_fullsized":"2024-04-25T18:46:33.424292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094777+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:46:33.094777+0000","last_peered":"2024-04-25T18:46:33.094777+0000","last_clean":"2024-04-25T18:46:33.094777+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:46:33.094777+0000","last_undegraded":"2024-04-25T18:46:33.094777+0000","last_fullsized":"2024-04-25T18:46:33.094777+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422751+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:46:33.422751+0000","last_peered":"2024-04-25T18:46:33.422751+0000","last_clean":"2024-04-25T18:46:33.422751+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:46:33.422751+0000","last_undegraded":"2024-04-25T18:46:33.422751+0000","last_fullsized":"2024-04-25T18:46:33.422751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892922+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:46:32.892922+0000","last_peered":"2024-04-25T18:46:32.892922+0000","last_clean":"2024-04-25T18:46:32.892922+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:46:32.892922+0000","last_undegraded":"2024-04-25T18:46:32.892922+0000","last_fullsized":"2024-04-25T18:46:32.892922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024639+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:46:33.024639+0000","last_peered":"2024-04-25T18:46:33.024639+0000","last_clean":"2024-04-25T18:46:33.024639+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:46:33.024639+0000","last_undegraded":"2024-04-25T18:46:33.024639+0000","last_fullsized":"2024-04-25T18:46:33.024639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936158+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:46:32.936158+0000","last_peered":"2024-04-25T18:46:32.936158+0000","last_clean":"2024-04-25T18:46:32.936158+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:46:32.936158+0000","last_undegraded":"2024-04-25T18:46:32.936158+0000","last_fullsized":"2024-04-25T18:46:32.936158+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889007+0000","last_change":"2024-04-25T18:43:55.436949+0000","last_active":"2024-04-25T18:46:32.889007+0000","last_peered":"2024-04-25T18:46:32.889007+0000","last_clean":"2024-04-25T18:46:32.889007+0000","last_became_active":"2024-04-25T18:43:55.435757+0000","last_became_peered":"2024-04-25T18:43:55.435757+0000","last_unstale":"2024-04-25T18:46:32.889007+0000","last_undegraded":"2024-04-25T18:46:32.889007+0000","last_fullsized":"2024-04-25T18:46:32.889007+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:46.736524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238289+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:46:33.238289+0000","last_peered":"2024-04-25T18:46:33.238289+0000","last_clean":"2024-04-25T18:46:33.238289+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:46:33.238289+0000","last_undegraded":"2024-04-25T18:46:33.238289+0000","last_fullsized":"2024-04-25T18:46:33.238289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095622+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:46:33.095622+0000","last_peered":"2024-04-25T18:46:33.095622+0000","last_clean":"2024-04-25T18:46:33.095622+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:46:33.095622+0000","last_undegraded":"2024-04-25T18:46:33.095622+0000","last_fullsized":"2024-04-25T18:46:33.095622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"96'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422615+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:46:33.422615+0000","last_peered":"2024-04-25T18:46:33.422615+0000","last_clean":"2024-04-25T18:46:33.422615+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:46:33.422615+0000","last_undegraded":"2024-04-25T18:46:33.422615+0000","last_fullsized":"2024-04-25T18:46:33.422615+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":597308,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1440,"num_read_kb":820,"num_write":294,"num_write_kb":1439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1791924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252931,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":523043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514568,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41420,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703732,"statfs":{"total":95995035648,"available":95952621568,"internally_reserved":0,"allocated":1908736,"data_stored":1314112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678094,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36784,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708368,"statfs":{"total":95995035648,"available":95957368832,"internally_reserved":0,"allocated":1478656,"data_stored":916383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939733,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36772,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708380,"statfs":{"total":95995035648,"available":95957381120,"internally_reserved":0,"allocated":1466368,"data_stored":915609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234073,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1662976,"data_stored":1106467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397599,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":522556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":589407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738499,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":514826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717008,"statfs":{"total":95995035648,"available":95966216192,"internally_reserved":0,"allocated":1150976,"data_stored":583908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117102,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":522867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":524729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659238,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32832,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712320,"statfs":{"total":95995035648,"available":95961415680,"internally_reserved":0,"allocated":1691648,"data_stored":1114310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":67058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":525100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":133858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":201278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:46:39.862 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:46:39.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:39.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:39.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:39 smithi070 ceph-mon[32820]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T18:46:40.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi070:/etc/ceph/ceph.conf 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi114:/etc/ceph/ceph.conf 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:40.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:39 smithi114 ceph-mon[32988]: Updating smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:40.291 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:40.792 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:40 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.792 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:40 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.792 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:40 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.793 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:40 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.793 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:40 smithi063 ceph-mon[28006]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:40 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:40 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:40 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:40 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:40.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:40 smithi070 ceph-mon[32820]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:40 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:40 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:40 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:40 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:40 smithi114 ceph-mon[32988]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T18:46:42.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:42.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T18:46:42.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.354 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.355 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:42.355 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:42.355 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:42.355 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T18:46:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:42.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:42.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:42.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 -- 172.21.15.63:0/3798840147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 msgr2=0x7f92a4115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:42.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 --2- 172.21.15.63:0/3798840147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a4115f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f929c00b0a0 tx=0x7f929c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 -- 172.21.15.63:0/3798840147 shutdown_connections 2024-04-25T18:46:42.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 --2- 172.21.15.63:0/3798840147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a4115f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 --2- 172.21.15.63:0/3798840147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a41096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 --2- 172.21.15.63:0/3798840147 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 0x7f92a410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 -- 172.21.15.63:0/3798840147 >> 172.21.15.63:0/3798840147 conn(0x7f92a4076210 msgr2=0x7f92a4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 -- 172.21.15.63:0/3798840147 shutdown_connections 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.601+0000 7f92ab600700 1 -- 172.21.15.63:0/3798840147 wait complete. 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.602+0000 7f92ab600700 1 Processor -- start 2024-04-25T18:46:42.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.602+0000 7f92ab600700 1 -- start start 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 0x7f92a419fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a41a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f92a41192c0 con 0x7f92a41092d0 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4119140 con 0x7f92a410e3b0 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92ab600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f92a4119440 con 0x7f92a4109c80 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92a8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92a8c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57772/0 (socket says 172.21.15.63:57772) 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92a9600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a41a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92a8c00700 1 -- 172.21.15.63:0/325724386 learned_addr learned my addr 172.21.15.63:0/325724386 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.603+0000 7f92a3e00700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 0x7f92a419fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:42.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 -- 172.21.15.63:0/325724386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 msgr2=0x7f92a419fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 0x7f92a419fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 -- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 msgr2=0x7f92a419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 -- 172.21.15.63:0/325724386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a41a4730 con 0x7f92a410e3b0 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a9600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a41a3ff0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f929c00b070 tx=0x7f929c004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:42.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c021070 con 0x7f92a410e3b0 2024-04-25T18:46:42.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f929c007b80 con 0x7f92a410e3b0 2024-04-25T18:46:42.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a41a49c0 con 0x7f92a410e3b0 2024-04-25T18:46:42.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c005040 con 0x7f92a410e3b0 2024-04-25T18:46:42.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.604+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a41ac2a0 con 0x7f92a410e3b0 2024-04-25T18:46:42.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.605+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a4069730 con 0x7f92a410e3b0 2024-04-25T18:46:42.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.608+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f929c024070 con 0x7f92a410e3b0 2024-04-25T18:46:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.609+0000 7f92a1600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 0x7f928c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.609+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f929c0d18a0 con 0x7f92a410e3b0 2024-04-25T18:46:42.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.609+0000 7f92a8c00700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 0x7f928c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:42.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.610+0000 7f92a8c00700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 0x7f928c0a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9298009d20 tx=0x7f9298009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:42.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.611+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f929c09ddf0 con 0x7f92a410e3b0 2024-04-25T18:46:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: Updating smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T18:46:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 131 KiB/s wr, 76 op/s 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:46:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:46:42.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.839+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92a4062e80 con 0x7f928c0a3ec0 2024-04-25T18:46:42.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.856+0000 7f92a1600700 1 -- 172.21.15.63:0/325724386 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257773 (secure 0 0 0) 0x7f92a4062e80 con 0x7f928c0a3ec0 2024-04-25T18:46:42.864 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:42.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.863+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 msgr2=0x7f928c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:42.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 0x7f928c0a6380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9298009d20 tx=0x7f9298009260 comp rx=0 tx=0).stop 2024-04-25T18:46:42.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 msgr2=0x7f92a41a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:42.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a41a3ff0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f929c00b070 tx=0x7f929c004c80 comp rx=0 tx=0).stop 2024-04-25T18:46:42.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 shutdown_connections 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f928c0a3ec0 0x7f928c0a6380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92a410e3b0 0x7f92a41a3ff0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92a41092d0 0x7f92a419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 --2- 172.21.15.63:0/325724386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f92a4109c80 0x7f92a419fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 >> 172.21.15.63:0/325724386 conn(0x7f92a4076210 msgr2=0x7f92a40767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 shutdown_connections 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:42.864+0000 7f92ab600700 1 -- 172.21.15.63:0/325724386 wait complete. 2024-04-25T18:46:42.867 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:46:43.808 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":7,"stamp":"2024-04-25T18:46:40.911229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1384542,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1963,"num_read_kb":1404,"num_write":623,"num_write_kb":5293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":959,"ondisk_log_size":959,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":407816,"kb_used_data":16800,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124534008,"statfs":{"total":1151940427776,"available":1151522824192,"internally_reserved":0,"allocated":17203200,"data_stored":10421814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":387,"num_read_kb":465,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.962899"},"pg_stats":[{"pgid":"3.3f","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422858+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:46:33.422858+0000","last_peered":"2024-04-25T18:46:33.422858+0000","last_clean":"2024-04-25T18:46:33.422858+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:46:33.422858+0000","last_undegraded":"2024-04-25T18:46:33.422858+0000","last_fullsized":"2024-04-25T18:46:33.422858+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"96'4","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424910+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:46:33.424910+0000","last_peered":"2024-04-25T18:46:33.424910+0000","last_clean":"2024-04-25T18:46:33.424910+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:46:33.424910+0000","last_undegraded":"2024-04-25T18:46:33.424910+0000","last_fullsized":"2024-04-25T18:46:33.424910+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"96'7","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424285+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:46:33.424285+0000","last_peered":"2024-04-25T18:46:33.424285+0000","last_clean":"2024-04-25T18:46:33.424285+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:46:33.424285+0000","last_undegraded":"2024-04-25T18:46:33.424285+0000","last_fullsized":"2024-04-25T18:46:33.424285+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890407+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:46:32.890407+0000","last_peered":"2024-04-25T18:46:32.890407+0000","last_clean":"2024-04-25T18:46:32.890407+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:46:32.890407+0000","last_undegraded":"2024-04-25T18:46:32.890407+0000","last_fullsized":"2024-04-25T18:46:32.890407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023708+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:46:33.023708+0000","last_peered":"2024-04-25T18:46:33.023708+0000","last_clean":"2024-04-25T18:46:33.023708+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:46:33.023708+0000","last_undegraded":"2024-04-25T18:46:33.023708+0000","last_fullsized":"2024-04-25T18:46:33.023708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933634+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:46:32.933634+0000","last_peered":"2024-04-25T18:46:32.933634+0000","last_clean":"2024-04-25T18:46:32.933634+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:46:32.933634+0000","last_undegraded":"2024-04-25T18:46:32.933634+0000","last_fullsized":"2024-04-25T18:46:32.933634+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231868+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:46:33.231868+0000","last_peered":"2024-04-25T18:46:33.231868+0000","last_clean":"2024-04-25T18:46:33.231868+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:46:33.231868+0000","last_undegraded":"2024-04-25T18:46:33.231868+0000","last_fullsized":"2024-04-25T18:46:33.231868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023428+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:46:33.023428+0000","last_peered":"2024-04-25T18:46:33.023428+0000","last_clean":"2024-04-25T18:46:33.023428+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:46:33.023428+0000","last_undegraded":"2024-04-25T18:46:33.023428+0000","last_fullsized":"2024-04-25T18:46:33.023428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238908+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:46:33.238908+0000","last_peered":"2024-04-25T18:46:33.238908+0000","last_clean":"2024-04-25T18:46:33.238908+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:46:33.238908+0000","last_undegraded":"2024-04-25T18:46:33.238908+0000","last_fullsized":"2024-04-25T18:46:33.238908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"96'16","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933875+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:46:32.933875+0000","last_peered":"2024-04-25T18:46:32.933875+0000","last_clean":"2024-04-25T18:46:32.933875+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:46:32.933875+0000","last_undegraded":"2024-04-25T18:46:32.933875+0000","last_fullsized":"2024-04-25T18:46:32.933875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232039+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:46:33.232039+0000","last_peered":"2024-04-25T18:46:33.232039+0000","last_clean":"2024-04-25T18:46:33.232039+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:46:33.232039+0000","last_undegraded":"2024-04-25T18:46:33.232039+0000","last_fullsized":"2024-04-25T18:46:33.232039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890594+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:46:32.890594+0000","last_peered":"2024-04-25T18:46:32.890594+0000","last_clean":"2024-04-25T18:46:32.890594+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:46:32.890594+0000","last_undegraded":"2024-04-25T18:46:32.890594+0000","last_fullsized":"2024-04-25T18:46:32.890594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095489+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:46:33.095489+0000","last_peered":"2024-04-25T18:46:33.095489+0000","last_clean":"2024-04-25T18:46:33.095489+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:46:33.095489+0000","last_undegraded":"2024-04-25T18:46:33.095489+0000","last_fullsized":"2024-04-25T18:46:33.095489+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423834+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:46:33.423834+0000","last_peered":"2024-04-25T18:46:33.423834+0000","last_clean":"2024-04-25T18:46:33.423834+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:46:33.423834+0000","last_undegraded":"2024-04-25T18:46:33.423834+0000","last_fullsized":"2024-04-25T18:46:33.423834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423857+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:46:33.423857+0000","last_peered":"2024-04-25T18:46:33.423857+0000","last_clean":"2024-04-25T18:46:33.423857+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:46:33.423857+0000","last_undegraded":"2024-04-25T18:46:33.423857+0000","last_fullsized":"2024-04-25T18:46:33.423857+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238084+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:46:33.238084+0000","last_peered":"2024-04-25T18:46:33.238084+0000","last_clean":"2024-04-25T18:46:33.238084+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:46:33.238084+0000","last_undegraded":"2024-04-25T18:46:33.238084+0000","last_fullsized":"2024-04-25T18:46:33.238084+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422975+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:46:33.422975+0000","last_peered":"2024-04-25T18:46:33.422975+0000","last_clean":"2024-04-25T18:46:33.422975+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:46:33.422975+0000","last_undegraded":"2024-04-25T18:46:33.422975+0000","last_fullsized":"2024-04-25T18:46:33.422975+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889201+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:46:32.889201+0000","last_peered":"2024-04-25T18:46:32.889201+0000","last_clean":"2024-04-25T18:46:32.889201+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:46:32.889201+0000","last_undegraded":"2024-04-25T18:46:32.889201+0000","last_fullsized":"2024-04-25T18:46:32.889201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892794+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:46:32.892794+0000","last_peered":"2024-04-25T18:46:32.892794+0000","last_clean":"2024-04-25T18:46:32.892794+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:46:32.892794+0000","last_undegraded":"2024-04-25T18:46:32.892794+0000","last_fullsized":"2024-04-25T18:46:32.892794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889557+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:46:32.889557+0000","last_peered":"2024-04-25T18:46:32.889557+0000","last_clean":"2024-04-25T18:46:32.889557+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:46:32.889557+0000","last_undegraded":"2024-04-25T18:46:32.889557+0000","last_fullsized":"2024-04-25T18:46:32.889557+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024156+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:46:33.024156+0000","last_peered":"2024-04-25T18:46:33.024156+0000","last_clean":"2024-04-25T18:46:33.024156+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:46:33.024156+0000","last_undegraded":"2024-04-25T18:46:33.024156+0000","last_fullsized":"2024-04-25T18:46:33.024156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024791+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:46:33.024791+0000","last_peered":"2024-04-25T18:46:33.024791+0000","last_clean":"2024-04-25T18:46:33.024791+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:46:33.024791+0000","last_undegraded":"2024-04-25T18:46:33.024791+0000","last_fullsized":"2024-04-25T18:46:33.024791+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"96'6","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424961+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:46:33.424961+0000","last_peered":"2024-04-25T18:46:33.424961+0000","last_clean":"2024-04-25T18:46:33.424961+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:46:33.424961+0000","last_undegraded":"2024-04-25T18:46:33.424961+0000","last_fullsized":"2024-04-25T18:46:33.424961+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024166+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:46:33.024166+0000","last_peered":"2024-04-25T18:46:33.024166+0000","last_clean":"2024-04-25T18:46:33.024166+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:46:33.024166+0000","last_undegraded":"2024-04-25T18:46:33.024166+0000","last_fullsized":"2024-04-25T18:46:33.024166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238451+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:46:33.238451+0000","last_peered":"2024-04-25T18:46:33.238451+0000","last_clean":"2024-04-25T18:46:33.238451+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:46:33.238451+0000","last_undegraded":"2024-04-25T18:46:33.238451+0000","last_fullsized":"2024-04-25T18:46:33.238451+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024005+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:46:33.024005+0000","last_peered":"2024-04-25T18:46:33.024005+0000","last_clean":"2024-04-25T18:46:33.024005+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:46:33.024005+0000","last_undegraded":"2024-04-25T18:46:33.024005+0000","last_fullsized":"2024-04-25T18:46:33.024005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"96'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936393+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:46:32.936393+0000","last_peered":"2024-04-25T18:46:32.936393+0000","last_clean":"2024-04-25T18:46:32.936393+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:46:32.936393+0000","last_undegraded":"2024-04-25T18:46:32.936393+0000","last_fullsized":"2024-04-25T18:46:32.936393+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"97'7","reported_seq":268,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:36.941795+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:46:36.941795+0000","last_peered":"2024-04-25T18:46:36.941795+0000","last_clean":"2024-04-25T18:46:36.941795+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:46:36.941795+0000","last_undegraded":"2024-04-25T18:46:36.941795+0000","last_fullsized":"2024-04-25T18:46:36.941795+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":228,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"96'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023638+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:46:33.023638+0000","last_peered":"2024-04-25T18:46:33.023638+0000","last_clean":"2024-04-25T18:46:33.023638+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:46:33.023638+0000","last_undegraded":"2024-04-25T18:46:33.023638+0000","last_fullsized":"2024-04-25T18:46:33.023638+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933556+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:46:32.933556+0000","last_peered":"2024-04-25T18:46:32.933556+0000","last_clean":"2024-04-25T18:46:32.933556+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:46:32.933556+0000","last_undegraded":"2024-04-25T18:46:32.933556+0000","last_fullsized":"2024-04-25T18:46:32.933556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936363+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:46:32.936363+0000","last_peered":"2024-04-25T18:46:32.936363+0000","last_clean":"2024-04-25T18:46:32.936363+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:46:32.936363+0000","last_undegraded":"2024-04-25T18:46:32.936363+0000","last_fullsized":"2024-04-25T18:46:32.936363+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095036+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:46:33.095036+0000","last_peered":"2024-04-25T18:46:33.095036+0000","last_clean":"2024-04-25T18:46:33.095036+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:46:33.095036+0000","last_undegraded":"2024-04-25T18:46:33.095036+0000","last_fullsized":"2024-04-25T18:46:33.095036+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023495+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:46:33.023495+0000","last_peered":"2024-04-25T18:46:33.023495+0000","last_clean":"2024-04-25T18:46:33.023495+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:46:33.023495+0000","last_undegraded":"2024-04-25T18:46:33.023495+0000","last_fullsized":"2024-04-25T18:46:33.023495+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936092+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:46:32.936092+0000","last_peered":"2024-04-25T18:46:32.936092+0000","last_clean":"2024-04-25T18:46:32.936092+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:46:32.936092+0000","last_undegraded":"2024-04-25T18:46:32.936092+0000","last_fullsized":"2024-04-25T18:46:32.936092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889094+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:46:32.889094+0000","last_peered":"2024-04-25T18:46:32.889094+0000","last_clean":"2024-04-25T18:46:32.889094+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:46:32.889094+0000","last_undegraded":"2024-04-25T18:46:32.889094+0000","last_fullsized":"2024-04-25T18:46:32.889094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":262,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:36.941391+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:46:36.941391+0000","last_peered":"2024-04-25T18:46:36.941391+0000","last_clean":"2024-04-25T18:46:36.941391+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:46:36.941391+0000","last_undegraded":"2024-04-25T18:46:36.941391+0000","last_fullsized":"2024-04-25T18:46:36.941391+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":225,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238717+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:46:33.238717+0000","last_peered":"2024-04-25T18:46:33.238717+0000","last_clean":"2024-04-25T18:46:33.238717+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:46:33.238717+0000","last_undegraded":"2024-04-25T18:46:33.238717+0000","last_fullsized":"2024-04-25T18:46:33.238717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422655+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:46:33.422655+0000","last_peered":"2024-04-25T18:46:33.422655+0000","last_clean":"2024-04-25T18:46:33.422655+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:46:33.422655+0000","last_undegraded":"2024-04-25T18:46:33.422655+0000","last_fullsized":"2024-04-25T18:46:33.422655+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.425055+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:46:33.425055+0000","last_peered":"2024-04-25T18:46:33.425055+0000","last_clean":"2024-04-25T18:46:33.425055+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:46:33.425055+0000","last_undegraded":"2024-04-25T18:46:33.425055+0000","last_fullsized":"2024-04-25T18:46:33.425055+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936480+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:46:32.936480+0000","last_peered":"2024-04-25T18:46:32.936480+0000","last_clean":"2024-04-25T18:46:32.936480+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:46:32.936480+0000","last_undegraded":"2024-04-25T18:46:32.936480+0000","last_fullsized":"2024-04-25T18:46:32.936480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"96'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238458+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:46:33.238458+0000","last_peered":"2024-04-25T18:46:33.238458+0000","last_clean":"2024-04-25T18:46:33.238458+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:46:33.238458+0000","last_undegraded":"2024-04-25T18:46:33.238458+0000","last_fullsized":"2024-04-25T18:46:33.238458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"97'36","reported_seq":531,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:37.942734+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:46:37.942734+0000","last_peered":"2024-04-25T18:46:37.942734+0000","last_clean":"2024-04-25T18:46:37.942734+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:46:37.942734+0000","last_undegraded":"2024-04-25T18:46:37.942734+0000","last_fullsized":"2024-04-25T18:46:37.942734+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66476,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":462,"num_read_kb":296,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023970+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:46:33.023970+0000","last_peered":"2024-04-25T18:46:33.023970+0000","last_clean":"2024-04-25T18:46:33.023970+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:46:33.023970+0000","last_undegraded":"2024-04-25T18:46:33.023970+0000","last_fullsized":"2024-04-25T18:46:33.023970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892434+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:46:32.892434+0000","last_peered":"2024-04-25T18:46:32.892434+0000","last_clean":"2024-04-25T18:46:32.892434+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:46:32.892434+0000","last_undegraded":"2024-04-25T18:46:32.892434+0000","last_fullsized":"2024-04-25T18:46:32.892434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094919+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:46:33.094919+0000","last_peered":"2024-04-25T18:46:33.094919+0000","last_clean":"2024-04-25T18:46:33.094919+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:46:33.094919+0000","last_undegraded":"2024-04-25T18:46:33.094919+0000","last_fullsized":"2024-04-25T18:46:33.094919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936173+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:46:32.936173+0000","last_peered":"2024-04-25T18:46:32.936173+0000","last_clean":"2024-04-25T18:46:32.936173+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:46:32.936173+0000","last_undegraded":"2024-04-25T18:46:32.936173+0000","last_fullsized":"2024-04-25T18:46:32.936173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232141+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:46:33.232141+0000","last_peered":"2024-04-25T18:46:33.232141+0000","last_clean":"2024-04-25T18:46:33.232141+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:46:33.232141+0000","last_undegraded":"2024-04-25T18:46:33.232141+0000","last_fullsized":"2024-04-25T18:46:33.232141+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"97'8","reported_seq":272,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:37.103026+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:46:37.103026+0000","last_peered":"2024-04-25T18:46:37.103026+0000","last_clean":"2024-04-25T18:46:37.103026+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:46:37.103026+0000","last_undegraded":"2024-04-25T18:46:37.103026+0000","last_fullsized":"2024-04-25T18:46:37.103026+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":232,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231813+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:46:33.231813+0000","last_peered":"2024-04-25T18:46:33.231813+0000","last_clean":"2024-04-25T18:46:33.231813+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:46:33.231813+0000","last_undegraded":"2024-04-25T18:46:33.231813+0000","last_fullsized":"2024-04-25T18:46:33.231813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422514+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:46:33.422514+0000","last_peered":"2024-04-25T18:46:33.422514+0000","last_clean":"2024-04-25T18:46:33.422514+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:46:33.422514+0000","last_undegraded":"2024-04-25T18:46:33.422514+0000","last_fullsized":"2024-04-25T18:46:33.422514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936373+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:46:32.936373+0000","last_peered":"2024-04-25T18:46:32.936373+0000","last_clean":"2024-04-25T18:46:32.936373+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:46:32.936373+0000","last_undegraded":"2024-04-25T18:46:32.936373+0000","last_fullsized":"2024-04-25T18:46:32.936373+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024089+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:46:33.024089+0000","last_peered":"2024-04-25T18:46:33.024089+0000","last_clean":"2024-04-25T18:46:33.024089+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:46:33.024089+0000","last_undegraded":"2024-04-25T18:46:33.024089+0000","last_fullsized":"2024-04-25T18:46:33.024089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422673+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:46:33.422673+0000","last_peered":"2024-04-25T18:46:33.422673+0000","last_clean":"2024-04-25T18:46:33.422673+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:46:33.422673+0000","last_undegraded":"2024-04-25T18:46:33.422673+0000","last_fullsized":"2024-04-25T18:46:33.422673+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889115+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:46:32.889115+0000","last_peered":"2024-04-25T18:46:32.889115+0000","last_clean":"2024-04-25T18:46:32.889115+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:46:32.889115+0000","last_undegraded":"2024-04-25T18:46:32.889115+0000","last_fullsized":"2024-04-25T18:46:32.889115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933754+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:46:32.933754+0000","last_peered":"2024-04-25T18:46:32.933754+0000","last_clean":"2024-04-25T18:46:32.933754+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:46:32.933754+0000","last_undegraded":"2024-04-25T18:46:32.933754+0000","last_fullsized":"2024-04-25T18:46:32.933754+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232202+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:46:33.232202+0000","last_peered":"2024-04-25T18:46:33.232202+0000","last_clean":"2024-04-25T18:46:33.232202+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:46:33.232202+0000","last_undegraded":"2024-04-25T18:46:33.232202+0000","last_fullsized":"2024-04-25T18:46:33.232202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933629+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:46:32.933629+0000","last_peered":"2024-04-25T18:46:32.933629+0000","last_clean":"2024-04-25T18:46:32.933629+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:46:32.933629+0000","last_undegraded":"2024-04-25T18:46:32.933629+0000","last_fullsized":"2024-04-25T18:46:32.933629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.888871+0000","last_change":"2024-04-25T18:43:55.423277+0000","last_active":"2024-04-25T18:46:32.888871+0000","last_peered":"2024-04-25T18:46:32.888871+0000","last_clean":"2024-04-25T18:46:32.888871+0000","last_became_active":"2024-04-25T18:43:55.421960+0000","last_became_peered":"2024-04-25T18:43:55.421960+0000","last_unstale":"2024-04-25T18:46:32.888871+0000","last_undegraded":"2024-04-25T18:46:32.888871+0000","last_fullsized":"2024-04-25T18:46:32.888871+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:27.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":265,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:37.102392+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:46:37.102392+0000","last_peered":"2024-04-25T18:46:37.102392+0000","last_clean":"2024-04-25T18:46:37.102392+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:46:37.102392+0000","last_undegraded":"2024-04-25T18:46:37.102392+0000","last_fullsized":"2024-04-25T18:46:37.102392+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":225,"num_read_kb":225,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024075+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:46:33.024075+0000","last_peered":"2024-04-25T18:46:33.024075+0000","last_clean":"2024-04-25T18:46:33.024075+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:46:33.024075+0000","last_undegraded":"2024-04-25T18:46:33.024075+0000","last_fullsized":"2024-04-25T18:46:33.024075+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890279+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:46:32.890279+0000","last_peered":"2024-04-25T18:46:32.890279+0000","last_clean":"2024-04-25T18:46:32.890279+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:46:32.890279+0000","last_undegraded":"2024-04-25T18:46:32.890279+0000","last_fullsized":"2024-04-25T18:46:32.890279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"97'148","reported_seq":413,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.036283+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:46:35.036283+0000","last_peered":"2024-04-25T18:46:35.036283+0000","last_clean":"2024-04-25T18:46:35.036283+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:46:35.036283+0000","last_undegraded":"2024-04-25T18:46:35.036283+0000","last_fullsized":"2024-04-25T18:46:35.036283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094794+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:46:33.094794+0000","last_peered":"2024-04-25T18:46:33.094794+0000","last_clean":"2024-04-25T18:46:33.094794+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:46:33.094794+0000","last_undegraded":"2024-04-25T18:46:33.094794+0000","last_fullsized":"2024-04-25T18:46:33.094794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232115+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:46:33.232115+0000","last_peered":"2024-04-25T18:46:33.232115+0000","last_clean":"2024-04-25T18:46:33.232115+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:46:33.232115+0000","last_undegraded":"2024-04-25T18:46:33.232115+0000","last_fullsized":"2024-04-25T18:46:33.232115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024110+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:46:33.024110+0000","last_peered":"2024-04-25T18:46:33.024110+0000","last_clean":"2024-04-25T18:46:33.024110+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:46:33.024110+0000","last_undegraded":"2024-04-25T18:46:33.024110+0000","last_fullsized":"2024-04-25T18:46:33.024110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094993+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:46:33.094993+0000","last_peered":"2024-04-25T18:46:33.094993+0000","last_clean":"2024-04-25T18:46:33.094993+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:46:33.094993+0000","last_undegraded":"2024-04-25T18:46:33.094993+0000","last_fullsized":"2024-04-25T18:46:33.094993+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"97'100","reported_seq":225,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.724020+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:46:35.724020+0000","last_peered":"2024-04-25T18:46:35.724020+0000","last_clean":"2024-04-25T18:46:35.724020+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:46:35.724020+0000","last_undegraded":"2024-04-25T18:46:35.724020+0000","last_fullsized":"2024-04-25T18:46:35.724020+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":184,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024773+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:46:33.024773+0000","last_peered":"2024-04-25T18:46:33.024773+0000","last_clean":"2024-04-25T18:46:33.024773+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:46:33.024773+0000","last_undegraded":"2024-04-25T18:46:33.024773+0000","last_fullsized":"2024-04-25T18:46:33.024773+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890406+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:46:32.890406+0000","last_peered":"2024-04-25T18:46:32.890406+0000","last_clean":"2024-04-25T18:46:32.890406+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:46:32.890406+0000","last_undegraded":"2024-04-25T18:46:32.890406+0000","last_fullsized":"2024-04-25T18:46:32.890406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424814+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:46:33.424814+0000","last_peered":"2024-04-25T18:46:33.424814+0000","last_clean":"2024-04-25T18:46:33.424814+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:46:33.424814+0000","last_undegraded":"2024-04-25T18:46:33.424814+0000","last_fullsized":"2024-04-25T18:46:33.424814+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238097+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:46:33.238097+0000","last_peered":"2024-04-25T18:46:33.238097+0000","last_clean":"2024-04-25T18:46:33.238097+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:46:33.238097+0000","last_undegraded":"2024-04-25T18:46:33.238097+0000","last_fullsized":"2024-04-25T18:46:33.238097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023626+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:46:33.023626+0000","last_peered":"2024-04-25T18:46:33.023626+0000","last_clean":"2024-04-25T18:46:33.023626+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:46:33.023626+0000","last_undegraded":"2024-04-25T18:46:33.023626+0000","last_fullsized":"2024-04-25T18:46:33.023626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023651+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:46:33.023651+0000","last_peered":"2024-04-25T18:46:33.023651+0000","last_clean":"2024-04-25T18:46:33.023651+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:46:33.023651+0000","last_undegraded":"2024-04-25T18:46:33.023651+0000","last_fullsized":"2024-04-25T18:46:33.023651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422592+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:46:33.422592+0000","last_peered":"2024-04-25T18:46:33.422592+0000","last_clean":"2024-04-25T18:46:33.422592+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:46:33.422592+0000","last_undegraded":"2024-04-25T18:46:33.422592+0000","last_fullsized":"2024-04-25T18:46:33.422592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892547+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:46:32.892547+0000","last_peered":"2024-04-25T18:46:32.892547+0000","last_clean":"2024-04-25T18:46:32.892547+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:46:32.892547+0000","last_undegraded":"2024-04-25T18:46:32.892547+0000","last_fullsized":"2024-04-25T18:46:32.892547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024105+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:46:33.024105+0000","last_peered":"2024-04-25T18:46:33.024105+0000","last_clean":"2024-04-25T18:46:33.024105+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:46:33.024105+0000","last_undegraded":"2024-04-25T18:46:33.024105+0000","last_fullsized":"2024-04-25T18:46:33.024105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023735+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:46:33.023735+0000","last_peered":"2024-04-25T18:46:33.023735+0000","last_clean":"2024-04-25T18:46:33.023735+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:46:33.023735+0000","last_undegraded":"2024-04-25T18:46:33.023735+0000","last_fullsized":"2024-04-25T18:46:33.023735+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231981+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:46:33.231981+0000","last_peered":"2024-04-25T18:46:33.231981+0000","last_clean":"2024-04-25T18:46:33.231981+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:46:33.231981+0000","last_undegraded":"2024-04-25T18:46:33.231981+0000","last_fullsized":"2024-04-25T18:46:33.231981+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892740+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:46:32.892740+0000","last_peered":"2024-04-25T18:46:32.892740+0000","last_clean":"2024-04-25T18:46:32.892740+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:46:32.892740+0000","last_undegraded":"2024-04-25T18:46:32.892740+0000","last_fullsized":"2024-04-25T18:46:32.892740+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232147+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:46:33.232147+0000","last_peered":"2024-04-25T18:46:33.232147+0000","last_clean":"2024-04-25T18:46:33.232147+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:46:33.232147+0000","last_undegraded":"2024-04-25T18:46:33.232147+0000","last_fullsized":"2024-04-25T18:46:33.232147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"97'65","reported_seq":129,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.205096+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:46:35.205096+0000","last_peered":"2024-04-25T18:46:35.205096+0000","last_clean":"2024-04-25T18:46:35.205096+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:46:35.205096+0000","last_undegraded":"2024-04-25T18:46:35.205096+0000","last_fullsized":"2024-04-25T18:46:35.205096+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232014+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:46:33.232014+0000","last_peered":"2024-04-25T18:46:33.232014+0000","last_clean":"2024-04-25T18:46:33.232014+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:46:33.232014+0000","last_undegraded":"2024-04-25T18:46:33.232014+0000","last_fullsized":"2024-04-25T18:46:33.232014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094761+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:46:33.094761+0000","last_peered":"2024-04-25T18:46:33.094761+0000","last_clean":"2024-04-25T18:46:33.094761+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:46:33.094761+0000","last_undegraded":"2024-04-25T18:46:33.094761+0000","last_fullsized":"2024-04-25T18:46:33.094761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232252+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:46:33.232252+0000","last_peered":"2024-04-25T18:46:33.232252+0000","last_clean":"2024-04-25T18:46:33.232252+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:46:33.232252+0000","last_undegraded":"2024-04-25T18:46:33.232252+0000","last_fullsized":"2024-04-25T18:46:33.232252+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"96'9","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422784+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:46:33.422784+0000","last_peered":"2024-04-25T18:46:33.422784+0000","last_clean":"2024-04-25T18:46:33.422784+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:46:33.422784+0000","last_undegraded":"2024-04-25T18:46:33.422784+0000","last_fullsized":"2024-04-25T18:46:33.422784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"96'12","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892408+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:46:32.892408+0000","last_peered":"2024-04-25T18:46:32.892408+0000","last_clean":"2024-04-25T18:46:32.892408+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:46:32.892408+0000","last_undegraded":"2024-04-25T18:46:32.892408+0000","last_fullsized":"2024-04-25T18:46:32.892408+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892321+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:46:32.892321+0000","last_peered":"2024-04-25T18:46:32.892321+0000","last_clean":"2024-04-25T18:46:32.892321+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:46:32.892321+0000","last_undegraded":"2024-04-25T18:46:32.892321+0000","last_fullsized":"2024-04-25T18:46:32.892321+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424292+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:46:33.424292+0000","last_peered":"2024-04-25T18:46:33.424292+0000","last_clean":"2024-04-25T18:46:33.424292+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:46:33.424292+0000","last_undegraded":"2024-04-25T18:46:33.424292+0000","last_fullsized":"2024-04-25T18:46:33.424292+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094777+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:46:33.094777+0000","last_peered":"2024-04-25T18:46:33.094777+0000","last_clean":"2024-04-25T18:46:33.094777+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:46:33.094777+0000","last_undegraded":"2024-04-25T18:46:33.094777+0000","last_fullsized":"2024-04-25T18:46:33.094777+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422751+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:46:33.422751+0000","last_peered":"2024-04-25T18:46:33.422751+0000","last_clean":"2024-04-25T18:46:33.422751+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:46:33.422751+0000","last_undegraded":"2024-04-25T18:46:33.422751+0000","last_fullsized":"2024-04-25T18:46:33.422751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892922+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:46:32.892922+0000","last_peered":"2024-04-25T18:46:32.892922+0000","last_clean":"2024-04-25T18:46:32.892922+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:46:32.892922+0000","last_undegraded":"2024-04-25T18:46:32.892922+0000","last_fullsized":"2024-04-25T18:46:32.892922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024639+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:46:33.024639+0000","last_peered":"2024-04-25T18:46:33.024639+0000","last_clean":"2024-04-25T18:46:33.024639+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:46:33.024639+0000","last_undegraded":"2024-04-25T18:46:33.024639+0000","last_fullsized":"2024-04-25T18:46:33.024639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936158+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:46:32.936158+0000","last_peered":"2024-04-25T18:46:32.936158+0000","last_clean":"2024-04-25T18:46:32.936158+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:46:32.936158+0000","last_undegraded":"2024-04-25T18:46:32.936158+0000","last_fullsized":"2024-04-25T18:46:32.936158+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889007+0000","last_change":"2024-04-25T18:43:55.436949+0000","last_active":"2024-04-25T18:46:32.889007+0000","last_peered":"2024-04-25T18:46:32.889007+0000","last_clean":"2024-04-25T18:46:32.889007+0000","last_became_active":"2024-04-25T18:43:55.435757+0000","last_became_peered":"2024-04-25T18:43:55.435757+0000","last_unstale":"2024-04-25T18:46:32.889007+0000","last_undegraded":"2024-04-25T18:46:32.889007+0000","last_fullsized":"2024-04-25T18:46:32.889007+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:46.736524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238289+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:46:33.238289+0000","last_peered":"2024-04-25T18:46:33.238289+0000","last_clean":"2024-04-25T18:46:33.238289+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:46:33.238289+0000","last_undegraded":"2024-04-25T18:46:33.238289+0000","last_fullsized":"2024-04-25T18:46:33.238289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095622+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:46:33.095622+0000","last_peered":"2024-04-25T18:46:33.095622+0000","last_clean":"2024-04-25T18:46:33.095622+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:46:33.095622+0000","last_undegraded":"2024-04-25T18:46:33.095622+0000","last_fullsized":"2024-04-25T18:46:33.095622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"96'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422615+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:46:33.422615+0000","last_peered":"2024-04-25T18:46:33.422615+0000","last_clean":"2024-04-25T18:46:33.422615+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:46:33.422615+0000","last_undegraded":"2024-04-25T18:46:33.422615+0000","last_fullsized":"2024-04-25T18:46:33.422615+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":793940,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1726,"num_read_kb":1074,"num_write":420,"num_write_kb":2353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2617344,"data_stored":2381820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252932,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":523043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514569,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41552,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703600,"statfs":{"total":95995035648,"available":95952486400,"internally_reserved":0,"allocated":2105344,"data_stored":1510744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678095,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36780,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708372,"statfs":{"total":95995035648,"available":95957372928,"internally_reserved":0,"allocated":1478656,"data_stored":916383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939734,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36768,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708384,"statfs":{"total":95995035648,"available":95957385216,"internally_reserved":0,"allocated":1466368,"data_stored":915609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234074,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36976,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708176,"statfs":{"total":95995035648,"available":95957172224,"internally_reserved":0,"allocated":1679360,"data_stored":1117032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397600,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":522556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":589407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":523827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":590845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117103,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37228,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707924,"statfs":{"total":95995035648,"available":95956914176,"internally_reserved":0,"allocated":1814528,"data_stored":1244315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855466,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":524729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659239,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37432,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707720,"statfs":{"total":95995035648,"available":95956705280,"internally_reserved":0,"allocated":2023424,"data_stored":1443324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":67058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":397910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:46:43.812 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T18:46:43.813 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T18:46:44.314 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:44 smithi114 ceph-mon[32988]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:44 smithi114 ceph-mon[32988]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T18:46:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:44 smithi063 ceph-mon[28006]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:44 smithi063 ceph-mon[28006]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T18:46:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:44 smithi070 ceph-mon[32820]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:44 smithi070 ceph-mon[32820]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 59 op/s 2024-04-25T18:46:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 -- 172.21.15.63:0/2601079262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c108a50 msgr2=0x7f591c108e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 --2- 172.21.15.63:0/2601079262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c108a50 0x7f591c108e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f590c009990 tx=0x7f590c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 -- 172.21.15.63:0/2601079262 shutdown_connections 2024-04-25T18:46:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 --2- 172.21.15.63:0/2601079262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c06bab0 0x7f591c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 --2- 172.21.15.63:0/2601079262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c108a50 0x7f591c108e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 --2- 172.21.15.63:0/2601079262 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c06b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 -- 172.21.15.63:0/2601079262 >> 172.21.15.63:0/2601079262 conn(0x7f591c0788f0 msgr2=0x7f591c078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 -- 172.21.15.63:0/2601079262 shutdown_connections 2024-04-25T18:46:46.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.521+0000 7f5923c00700 1 -- 172.21.15.63:0/2601079262 wait complete. 2024-04-25T18:46:46.524 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:46 smithi063 ceph-mon[28006]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 Processor -- start 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 -- start start 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c19fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c108a50 0x7f591c10daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f591c112c30 con 0x7f591c108a50 2024-04-25T18:46:46.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f591c112ab0 con 0x7f591c06bab0 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.523+0000 7f5923c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f591c112db0 con 0x7f591c06b110 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5921200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c19fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43364/0 (socket says 172.21.15.63:43364) 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 -- 172.21.15.63:0/1972634593 learned_addr learned my addr 172.21.15.63:0/1972634593 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:46.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5921c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c108a50 0x7f591c10daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 -- 172.21.15.63:0/1972634593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 msgr2=0x7f591c19fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c19fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 -- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c108a50 msgr2=0x7f591c10daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c108a50 0x7f591c10daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5913e00700 1 -- 172.21.15.63:0/1972634593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c10e3f0 con 0x7f591c06bab0 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.524+0000 7f5921200700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c19fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f5913e00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f590400e9a0 tx=0x7f590400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590400ca10 con 0x7f591c06bab0 2024-04-25T18:46:46.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c10e6e0 con 0x7f591c06bab0 2024-04-25T18:46:46.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f590400cbb0 con 0x7f591c06bab0 2024-04-25T18:46:46.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904018680 con 0x7f591c06bab0 2024-04-25T18:46:46.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.525+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c19baa0 con 0x7f591c06bab0 2024-04-25T18:46:46.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.526+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c06a9c0 con 0x7f591c06bab0 2024-04-25T18:46:46.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.530+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f5904010430 con 0x7f591c06bab0 2024-04-25T18:46:46.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.531+0000 7f591a000700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 0x7f59000a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:46.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.531+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f59040d0bc0 con 0x7f591c06bab0 2024-04-25T18:46:46.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.532+0000 7f5921200700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 0x7f59000a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:46.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.532+0000 7f5921200700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 0x7f59000a6300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f590c006010 tx=0x7f590c00b260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.536+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f590409d080 con 0x7f591c06bab0 2024-04-25T18:46:46.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:46 smithi114 ceph-mon[32988]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-25T18:46:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:46 smithi070 ceph-mon[32820]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 83 KiB/s wr, 51 op/s 2024-04-25T18:46:46.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.765+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f591c19bc70 con 0x7f59000a3e40 2024-04-25T18:46:46.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.777+0000 7f591a000700 1 -- 172.21.15.63:0/1972634593 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+257774 (secure 0 0 0) 0x7f591c19bc70 con 0x7f59000a3e40 2024-04-25T18:46:46.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 msgr2=0x7f59000a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 0x7f59000a6300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f590c006010 tx=0x7f590c00b260 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 msgr2=0x7f591c1a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f590400e9a0 tx=0x7f590400ecb0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 shutdown_connections 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59000a3e40 0x7f59000a6300 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f591c06b110 0x7f591c19fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f591c108a50 0x7f591c10daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 --2- 172.21.15.63:0/1972634593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f591c06bab0 0x7f591c1a0380 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 >> 172.21.15.63:0/1972634593 conn(0x7f591c0788f0 msgr2=0x7f591c1067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 shutdown_connections 2024-04-25T18:46:46.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:46.783+0000 7f5923c00700 1 -- 172.21.15.63:0/1972634593 wait complete. 2024-04-25T18:46:46.786 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-25T18:46:47.814 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-04-25T18:46:44.912282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1384542,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1993,"num_read_kb":1419,"num_write":623,"num_write_kb":5293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1181430,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":959,"ondisk_log_size":959,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":407816,"kb_used_data":16800,"kb_used_omap":18,"kb_used_meta":390637,"kb_avail":1124534008,"statfs":{"total":1151940427776,"available":1151522824192,"internally_reserved":0,"allocated":17203200,"data_stored":10421814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":400012601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":417,"num_read_kb":480,"num_write":146,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.963952"},"pg_stats":[{"pgid":"3.3f","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423008+0000","last_change":"2024-04-25T18:42:28.894325+0000","last_active":"2024-04-25T18:46:33.423008+0000","last_peered":"2024-04-25T18:46:33.423008+0000","last_clean":"2024-04-25T18:46:33.423008+0000","last_became_active":"2024-04-25T18:42:28.893981+0000","last_became_peered":"2024-04-25T18:42:28.893981+0000","last_unstale":"2024-04-25T18:46:33.423008+0000","last_undegraded":"2024-04-25T18:46:33.423008+0000","last_fullsized":"2024-04-25T18:46:33.423008+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:21:06.217085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"96'4","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424910+0000","last_change":"2024-04-25T18:42:28.903786+0000","last_active":"2024-04-25T18:46:33.424910+0000","last_peered":"2024-04-25T18:46:33.424910+0000","last_clean":"2024-04-25T18:46:33.424910+0000","last_became_active":"2024-04-25T18:42:28.901789+0000","last_became_peered":"2024-04-25T18:42:28.901789+0000","last_unstale":"2024-04-25T18:46:33.424910+0000","last_undegraded":"2024-04-25T18:46:33.424910+0000","last_fullsized":"2024-04-25T18:46:33.424910+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:49:08.989621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"96'7","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424285+0000","last_change":"2024-04-25T18:42:28.903698+0000","last_active":"2024-04-25T18:46:33.424285+0000","last_peered":"2024-04-25T18:46:33.424285+0000","last_clean":"2024-04-25T18:46:33.424285+0000","last_became_active":"2024-04-25T18:42:28.900354+0000","last_became_peered":"2024-04-25T18:42:28.900354+0000","last_unstale":"2024-04-25T18:46:33.424285+0000","last_undegraded":"2024-04-25T18:46:33.424285+0000","last_fullsized":"2024-04-25T18:46:33.424285+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:43:31.994486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890407+0000","last_change":"2024-04-25T18:42:28.885126+0000","last_active":"2024-04-25T18:46:32.890407+0000","last_peered":"2024-04-25T18:46:32.890407+0000","last_clean":"2024-04-25T18:46:32.890407+0000","last_became_active":"2024-04-25T18:42:28.884713+0000","last_became_peered":"2024-04-25T18:42:28.884713+0000","last_unstale":"2024-04-25T18:46:32.890407+0000","last_undegraded":"2024-04-25T18:46:32.890407+0000","last_fullsized":"2024-04-25T18:46:32.890407+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:41:06.754355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023708+0000","last_change":"2024-04-25T18:42:28.905170+0000","last_active":"2024-04-25T18:46:33.023708+0000","last_peered":"2024-04-25T18:46:33.023708+0000","last_clean":"2024-04-25T18:46:33.023708+0000","last_became_active":"2024-04-25T18:42:28.904808+0000","last_became_peered":"2024-04-25T18:42:28.904808+0000","last_unstale":"2024-04-25T18:46:33.023708+0000","last_undegraded":"2024-04-25T18:46:33.023708+0000","last_fullsized":"2024-04-25T18:46:33.023708+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:39:40.887550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933634+0000","last_change":"2024-04-25T18:42:28.896801+0000","last_active":"2024-04-25T18:46:32.933634+0000","last_peered":"2024-04-25T18:46:32.933634+0000","last_clean":"2024-04-25T18:46:32.933634+0000","last_became_active":"2024-04-25T18:42:28.896398+0000","last_became_peered":"2024-04-25T18:42:28.896398+0000","last_unstale":"2024-04-25T18:46:32.933634+0000","last_undegraded":"2024-04-25T18:46:32.933634+0000","last_fullsized":"2024-04-25T18:46:32.933634+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:47:48.269524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231868+0000","last_change":"2024-04-25T18:42:28.895188+0000","last_active":"2024-04-25T18:46:33.231868+0000","last_peered":"2024-04-25T18:46:33.231868+0000","last_clean":"2024-04-25T18:46:33.231868+0000","last_became_active":"2024-04-25T18:42:28.894839+0000","last_became_peered":"2024-04-25T18:42:28.894839+0000","last_unstale":"2024-04-25T18:46:33.231868+0000","last_undegraded":"2024-04-25T18:46:33.231868+0000","last_fullsized":"2024-04-25T18:46:33.231868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:07:49.951509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023428+0000","last_change":"2024-04-25T18:42:28.898259+0000","last_active":"2024-04-25T18:46:33.023428+0000","last_peered":"2024-04-25T18:46:33.023428+0000","last_clean":"2024-04-25T18:46:33.023428+0000","last_became_active":"2024-04-25T18:42:28.895345+0000","last_became_peered":"2024-04-25T18:42:28.895345+0000","last_unstale":"2024-04-25T18:46:33.023428+0000","last_undegraded":"2024-04-25T18:46:33.023428+0000","last_fullsized":"2024-04-25T18:46:33.023428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:06.001565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238908+0000","last_change":"2024-04-25T18:42:28.875772+0000","last_active":"2024-04-25T18:46:33.238908+0000","last_peered":"2024-04-25T18:46:33.238908+0000","last_clean":"2024-04-25T18:46:33.238908+0000","last_became_active":"2024-04-25T18:42:28.875411+0000","last_became_peered":"2024-04-25T18:42:28.875411+0000","last_unstale":"2024-04-25T18:46:33.238908+0000","last_undegraded":"2024-04-25T18:46:33.238908+0000","last_fullsized":"2024-04-25T18:46:33.238908+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:49:45.589707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"96'16","reported_seq":43,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933875+0000","last_change":"2024-04-25T18:42:28.884491+0000","last_active":"2024-04-25T18:46:32.933875+0000","last_peered":"2024-04-25T18:46:32.933875+0000","last_clean":"2024-04-25T18:46:32.933875+0000","last_became_active":"2024-04-25T18:42:28.884173+0000","last_became_peered":"2024-04-25T18:42:28.884173+0000","last_unstale":"2024-04-25T18:46:32.933875+0000","last_undegraded":"2024-04-25T18:46:32.933875+0000","last_fullsized":"2024-04-25T18:46:32.933875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:07.479994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232039+0000","last_change":"2024-04-25T18:42:28.882562+0000","last_active":"2024-04-25T18:46:33.232039+0000","last_peered":"2024-04-25T18:46:33.232039+0000","last_clean":"2024-04-25T18:46:33.232039+0000","last_became_active":"2024-04-25T18:42:28.882164+0000","last_became_peered":"2024-04-25T18:42:28.882164+0000","last_unstale":"2024-04-25T18:46:33.232039+0000","last_undegraded":"2024-04-25T18:46:33.232039+0000","last_fullsized":"2024-04-25T18:46:33.232039+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:00:40.891533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890594+0000","last_change":"2024-04-25T18:42:28.883056+0000","last_active":"2024-04-25T18:46:32.890594+0000","last_peered":"2024-04-25T18:46:32.890594+0000","last_clean":"2024-04-25T18:46:32.890594+0000","last_became_active":"2024-04-25T18:42:28.882706+0000","last_became_peered":"2024-04-25T18:42:28.882706+0000","last_unstale":"2024-04-25T18:46:32.890594+0000","last_undegraded":"2024-04-25T18:46:32.890594+0000","last_fullsized":"2024-04-25T18:46:32.890594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.340765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095489+0000","last_change":"2024-04-25T18:42:28.883283+0000","last_active":"2024-04-25T18:46:33.095489+0000","last_peered":"2024-04-25T18:46:33.095489+0000","last_clean":"2024-04-25T18:46:33.095489+0000","last_became_active":"2024-04-25T18:42:28.881473+0000","last_became_peered":"2024-04-25T18:42:28.881473+0000","last_unstale":"2024-04-25T18:46:33.095489+0000","last_undegraded":"2024-04-25T18:46:33.095489+0000","last_fullsized":"2024-04-25T18:46:33.095489+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:26:09.044399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423834+0000","last_change":"2024-04-25T18:42:28.899609+0000","last_active":"2024-04-25T18:46:33.423834+0000","last_peered":"2024-04-25T18:46:33.423834+0000","last_clean":"2024-04-25T18:46:33.423834+0000","last_became_active":"2024-04-25T18:42:28.899222+0000","last_became_peered":"2024-04-25T18:42:28.899222+0000","last_unstale":"2024-04-25T18:46:33.423834+0000","last_undegraded":"2024-04-25T18:46:33.423834+0000","last_fullsized":"2024-04-25T18:46:33.423834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:43:02.840213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.423857+0000","last_change":"2024-04-25T18:42:28.894558+0000","last_active":"2024-04-25T18:46:33.423857+0000","last_peered":"2024-04-25T18:46:33.423857+0000","last_clean":"2024-04-25T18:46:33.423857+0000","last_became_active":"2024-04-25T18:42:28.894122+0000","last_became_peered":"2024-04-25T18:42:28.894122+0000","last_unstale":"2024-04-25T18:46:33.423857+0000","last_undegraded":"2024-04-25T18:46:33.423857+0000","last_fullsized":"2024-04-25T18:46:33.423857+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:37:16.442550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238084+0000","last_change":"2024-04-25T18:42:28.894925+0000","last_active":"2024-04-25T18:46:33.238084+0000","last_peered":"2024-04-25T18:46:33.238084+0000","last_clean":"2024-04-25T18:46:33.238084+0000","last_became_active":"2024-04-25T18:42:28.894101+0000","last_became_peered":"2024-04-25T18:42:28.894101+0000","last_unstale":"2024-04-25T18:46:33.238084+0000","last_undegraded":"2024-04-25T18:46:33.238084+0000","last_fullsized":"2024-04-25T18:46:33.238084+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:27:29.988495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422975+0000","last_change":"2024-04-25T18:42:28.895083+0000","last_active":"2024-04-25T18:46:33.422975+0000","last_peered":"2024-04-25T18:46:33.422975+0000","last_clean":"2024-04-25T18:46:33.422975+0000","last_became_active":"2024-04-25T18:42:28.894739+0000","last_became_peered":"2024-04-25T18:42:28.894739+0000","last_unstale":"2024-04-25T18:46:33.422975+0000","last_undegraded":"2024-04-25T18:46:33.422975+0000","last_fullsized":"2024-04-25T18:46:33.422975+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:43:30.047737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889201+0000","last_change":"2024-04-25T18:42:28.894439+0000","last_active":"2024-04-25T18:46:32.889201+0000","last_peered":"2024-04-25T18:46:32.889201+0000","last_clean":"2024-04-25T18:46:32.889201+0000","last_became_active":"2024-04-25T18:42:28.894035+0000","last_became_peered":"2024-04-25T18:42:28.894035+0000","last_unstale":"2024-04-25T18:46:32.889201+0000","last_undegraded":"2024-04-25T18:46:32.889201+0000","last_fullsized":"2024-04-25T18:46:32.889201+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:22:46.440621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892794+0000","last_change":"2024-04-25T18:42:28.899142+0000","last_active":"2024-04-25T18:46:32.892794+0000","last_peered":"2024-04-25T18:46:32.892794+0000","last_clean":"2024-04-25T18:46:32.892794+0000","last_became_active":"2024-04-25T18:42:28.898799+0000","last_became_peered":"2024-04-25T18:42:28.898799+0000","last_unstale":"2024-04-25T18:46:32.892794+0000","last_undegraded":"2024-04-25T18:46:32.892794+0000","last_fullsized":"2024-04-25T18:46:32.892794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:23.738841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"92'9","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889557+0000","last_change":"2024-04-25T18:42:54.699790+0000","last_active":"2024-04-25T18:46:32.889557+0000","last_peered":"2024-04-25T18:46:32.889557+0000","last_clean":"2024-04-25T18:46:32.889557+0000","last_became_active":"2024-04-25T18:42:54.685384+0000","last_became_peered":"2024-04-25T18:42:54.685384+0000","last_unstale":"2024-04-25T18:46:32.889557+0000","last_undegraded":"2024-04-25T18:46:32.889557+0000","last_fullsized":"2024-04-25T18:46:32.889557+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:01:41.710754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024156+0000","last_change":"2024-04-25T18:42:28.909580+0000","last_active":"2024-04-25T18:46:33.024156+0000","last_peered":"2024-04-25T18:46:33.024156+0000","last_clean":"2024-04-25T18:46:33.024156+0000","last_became_active":"2024-04-25T18:42:28.907481+0000","last_became_peered":"2024-04-25T18:42:28.907481+0000","last_unstale":"2024-04-25T18:46:33.024156+0000","last_undegraded":"2024-04-25T18:46:33.024156+0000","last_fullsized":"2024-04-25T18:46:33.024156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:35:35.389215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"92'2","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024791+0000","last_change":"2024-04-25T18:42:24.847826+0000","last_active":"2024-04-25T18:46:33.024791+0000","last_peered":"2024-04-25T18:46:33.024791+0000","last_clean":"2024-04-25T18:46:33.024791+0000","last_became_active":"2024-04-25T18:42:24.846929+0000","last_became_peered":"2024-04-25T18:42:24.846929+0000","last_unstale":"2024-04-25T18:46:33.024791+0000","last_undegraded":"2024-04-25T18:46:33.024791+0000","last_fullsized":"2024-04-25T18:46:33.024791+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:30.403227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"96'6","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424961+0000","last_change":"2024-04-25T18:42:28.904065+0000","last_active":"2024-04-25T18:46:33.424961+0000","last_peered":"2024-04-25T18:46:33.424961+0000","last_clean":"2024-04-25T18:46:33.424961+0000","last_became_active":"2024-04-25T18:42:28.903665+0000","last_became_peered":"2024-04-25T18:42:28.903665+0000","last_unstale":"2024-04-25T18:46:33.424961+0000","last_undegraded":"2024-04-25T18:46:33.424961+0000","last_fullsized":"2024-04-25T18:46:33.424961+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:46:37.641342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024166+0000","last_change":"2024-04-25T18:42:24.835210+0000","last_active":"2024-04-25T18:46:33.024166+0000","last_peered":"2024-04-25T18:46:33.024166+0000","last_clean":"2024-04-25T18:46:33.024166+0000","last_became_active":"2024-04-25T18:42:24.834788+0000","last_became_peered":"2024-04-25T18:42:24.834788+0000","last_unstale":"2024-04-25T18:46:33.024166+0000","last_undegraded":"2024-04-25T18:46:33.024166+0000","last_fullsized":"2024-04-25T18:46:33.024166+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:45.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238451+0000","last_change":"2024-04-25T18:42:28.879551+0000","last_active":"2024-04-25T18:46:33.238451+0000","last_peered":"2024-04-25T18:46:33.238451+0000","last_clean":"2024-04-25T18:46:33.238451+0000","last_became_active":"2024-04-25T18:42:28.879078+0000","last_became_peered":"2024-04-25T18:42:28.879078+0000","last_unstale":"2024-04-25T18:46:33.238451+0000","last_undegraded":"2024-04-25T18:46:33.238451+0000","last_fullsized":"2024-04-25T18:46:33.238451+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:30.798897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024005+0000","last_change":"2024-04-25T18:42:24.832430+0000","last_active":"2024-04-25T18:46:33.024005+0000","last_peered":"2024-04-25T18:46:33.024005+0000","last_clean":"2024-04-25T18:46:33.024005+0000","last_became_active":"2024-04-25T18:42:24.831882+0000","last_became_peered":"2024-04-25T18:42:24.831882+0000","last_unstale":"2024-04-25T18:46:33.024005+0000","last_undegraded":"2024-04-25T18:46:33.024005+0000","last_fullsized":"2024-04-25T18:46:33.024005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:54:30.867071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"96'15","reported_seq":48,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936393+0000","last_change":"2024-04-25T18:42:54.681497+0000","last_active":"2024-04-25T18:46:32.936393+0000","last_peered":"2024-04-25T18:46:32.936393+0000","last_clean":"2024-04-25T18:46:32.936393+0000","last_became_active":"2024-04-25T18:42:54.681067+0000","last_became_peered":"2024-04-25T18:42:54.681067+0000","last_unstale":"2024-04-25T18:46:32.936393+0000","last_undegraded":"2024-04-25T18:46:32.936393+0000","last_fullsized":"2024-04-25T18:46:32.936393+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:25:24.631866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"97'7","reported_seq":273,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:41.949097+0000","last_change":"2024-04-25T18:42:24.833947+0000","last_active":"2024-04-25T18:46:41.949097+0000","last_peered":"2024-04-25T18:46:41.949097+0000","last_clean":"2024-04-25T18:46:41.949097+0000","last_became_active":"2024-04-25T18:42:24.833593+0000","last_became_peered":"2024-04-25T18:42:24.833593+0000","last_unstale":"2024-04-25T18:46:41.949097+0000","last_undegraded":"2024-04-25T18:46:41.949097+0000","last_fullsized":"2024-04-25T18:46:41.949097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:55:07.383050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1404,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"96'7","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023638+0000","last_change":"2024-04-25T18:42:55.384178+0000","last_active":"2024-04-25T18:46:33.023638+0000","last_peered":"2024-04-25T18:46:33.023638+0000","last_clean":"2024-04-25T18:46:33.023638+0000","last_became_active":"2024-04-25T18:42:55.383830+0000","last_became_peered":"2024-04-25T18:42:55.383830+0000","last_unstale":"2024-04-25T18:46:33.023638+0000","last_undegraded":"2024-04-25T18:46:33.023638+0000","last_fullsized":"2024-04-25T18:46:33.023638+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:12.939469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933556+0000","last_change":"2024-04-25T18:42:24.837345+0000","last_active":"2024-04-25T18:46:32.933556+0000","last_peered":"2024-04-25T18:46:32.933556+0000","last_clean":"2024-04-25T18:46:32.933556+0000","last_became_active":"2024-04-25T18:42:24.836987+0000","last_became_peered":"2024-04-25T18:42:24.836987+0000","last_unstale":"2024-04-25T18:46:32.933556+0000","last_undegraded":"2024-04-25T18:46:32.933556+0000","last_fullsized":"2024-04-25T18:46:32.933556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:27:52.032055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936363+0000","last_change":"2024-04-25T18:42:28.897705+0000","last_active":"2024-04-25T18:46:32.936363+0000","last_peered":"2024-04-25T18:46:32.936363+0000","last_clean":"2024-04-25T18:46:32.936363+0000","last_became_active":"2024-04-25T18:42:28.897262+0000","last_became_peered":"2024-04-25T18:42:28.897262+0000","last_unstale":"2024-04-25T18:46:32.936363+0000","last_undegraded":"2024-04-25T18:46:32.936363+0000","last_fullsized":"2024-04-25T18:46:32.936363+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:51:19.151702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095036+0000","last_change":"2024-04-25T18:42:24.845106+0000","last_active":"2024-04-25T18:46:33.095036+0000","last_peered":"2024-04-25T18:46:33.095036+0000","last_clean":"2024-04-25T18:46:33.095036+0000","last_became_active":"2024-04-25T18:42:24.844339+0000","last_became_peered":"2024-04-25T18:42:24.844339+0000","last_unstale":"2024-04-25T18:46:33.095036+0000","last_undegraded":"2024-04-25T18:46:33.095036+0000","last_fullsized":"2024-04-25T18:46:33.095036+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:52.527882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023495+0000","last_change":"2024-04-25T18:42:28.903085+0000","last_active":"2024-04-25T18:46:33.023495+0000","last_peered":"2024-04-25T18:46:33.023495+0000","last_clean":"2024-04-25T18:46:33.023495+0000","last_became_active":"2024-04-25T18:42:28.901209+0000","last_became_peered":"2024-04-25T18:42:28.901209+0000","last_unstale":"2024-04-25T18:46:33.023495+0000","last_undegraded":"2024-04-25T18:46:33.023495+0000","last_fullsized":"2024-04-25T18:46:33.023495+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:56:41.713258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936092+0000","last_change":"2024-04-25T18:42:24.833204+0000","last_active":"2024-04-25T18:46:32.936092+0000","last_peered":"2024-04-25T18:46:32.936092+0000","last_clean":"2024-04-25T18:46:32.936092+0000","last_became_active":"2024-04-25T18:42:24.832822+0000","last_became_peered":"2024-04-25T18:42:24.832822+0000","last_unstale":"2024-04-25T18:46:32.936092+0000","last_undegraded":"2024-04-25T18:46:32.936092+0000","last_fullsized":"2024-04-25T18:46:32.936092+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:42.911829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889094+0000","last_change":"2024-04-25T18:42:28.910847+0000","last_active":"2024-04-25T18:46:32.889094+0000","last_peered":"2024-04-25T18:46:32.889094+0000","last_clean":"2024-04-25T18:46:32.889094+0000","last_became_active":"2024-04-25T18:42:28.910454+0000","last_became_peered":"2024-04-25T18:42:28.910454+0000","last_unstale":"2024-04-25T18:46:32.889094+0000","last_undegraded":"2024-04-25T18:46:32.889094+0000","last_fullsized":"2024-04-25T18:46:32.889094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:45:15.793454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"97'4","reported_seq":267,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:41.948513+0000","last_change":"2024-04-25T18:42:24.840779+0000","last_active":"2024-04-25T18:46:41.948513+0000","last_peered":"2024-04-25T18:46:41.948513+0000","last_clean":"2024-04-25T18:46:41.948513+0000","last_became_active":"2024-04-25T18:42:24.840474+0000","last_became_peered":"2024-04-25T18:42:24.840474+0000","last_unstale":"2024-04-25T18:46:41.948513+0000","last_undegraded":"2024-04-25T18:46:41.948513+0000","last_fullsized":"2024-04-25T18:46:41.948513+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:16.272241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":230,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238717+0000","last_change":"2024-04-25T18:42:28.895894+0000","last_active":"2024-04-25T18:46:33.238717+0000","last_peered":"2024-04-25T18:46:33.238717+0000","last_clean":"2024-04-25T18:46:33.238717+0000","last_became_active":"2024-04-25T18:42:28.895539+0000","last_became_peered":"2024-04-25T18:42:28.895539+0000","last_unstale":"2024-04-25T18:46:33.238717+0000","last_undegraded":"2024-04-25T18:46:33.238717+0000","last_fullsized":"2024-04-25T18:46:33.238717+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:56.484836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"92'2","reported_seq":23,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422655+0000","last_change":"2024-04-25T18:42:55.117574+0000","last_active":"2024-04-25T18:46:33.422655+0000","last_peered":"2024-04-25T18:46:33.422655+0000","last_clean":"2024-04-25T18:46:33.422655+0000","last_became_active":"2024-04-25T18:42:55.113627+0000","last_became_peered":"2024-04-25T18:42:55.113627+0000","last_unstale":"2024-04-25T18:46:33.422655+0000","last_undegraded":"2024-04-25T18:46:33.422655+0000","last_fullsized":"2024-04-25T18:46:33.422655+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:15.625848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.17","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.425055+0000","last_change":"2024-04-25T18:42:28.903178+0000","last_active":"2024-04-25T18:46:33.425055+0000","last_peered":"2024-04-25T18:46:33.425055+0000","last_clean":"2024-04-25T18:46:33.425055+0000","last_became_active":"2024-04-25T18:42:28.899350+0000","last_became_peered":"2024-04-25T18:42:28.899350+0000","last_unstale":"2024-04-25T18:46:33.425055+0000","last_undegraded":"2024-04-25T18:46:33.425055+0000","last_fullsized":"2024-04-25T18:46:33.425055+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:10.716331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"92'4","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936480+0000","last_change":"2024-04-25T18:42:24.849310+0000","last_active":"2024-04-25T18:46:32.936480+0000","last_peered":"2024-04-25T18:46:32.936480+0000","last_clean":"2024-04-25T18:46:32.936480+0000","last_became_active":"2024-04-25T18:42:24.848957+0000","last_became_peered":"2024-04-25T18:42:24.848957+0000","last_unstale":"2024-04-25T18:46:32.936480+0000","last_undegraded":"2024-04-25T18:46:32.936480+0000","last_fullsized":"2024-04-25T18:46:32.936480+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:33.538463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"96'11","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238458+0000","last_change":"2024-04-25T18:42:54.680465+0000","last_active":"2024-04-25T18:46:33.238458+0000","last_peered":"2024-04-25T18:46:33.238458+0000","last_clean":"2024-04-25T18:46:33.238458+0000","last_became_active":"2024-04-25T18:42:54.680093+0000","last_became_peered":"2024-04-25T18:42:54.680093+0000","last_unstale":"2024-04-25T18:46:33.238458+0000","last_undegraded":"2024-04-25T18:46:33.238458+0000","last_fullsized":"2024-04-25T18:46:33.238458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:02.912167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,7],"acting":[1,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"97'36","reported_seq":541,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:42.950139+0000","last_change":"2024-04-25T18:42:24.833332+0000","last_active":"2024-04-25T18:46:42.950139+0000","last_peered":"2024-04-25T18:46:42.950139+0000","last_clean":"2024-04-25T18:46:42.950139+0000","last_became_active":"2024-04-25T18:42:24.833014+0000","last_became_peered":"2024-04-25T18:42:24.833014+0000","last_unstale":"2024-04-25T18:46:42.950139+0000","last_undegraded":"2024-04-25T18:46:42.950139+0000","last_fullsized":"2024-04-25T18:46:42.950139+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:22:32.545171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66476,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":472,"num_read_kb":301,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023970+0000","last_change":"2024-04-25T18:42:28.887583+0000","last_active":"2024-04-25T18:46:33.023970+0000","last_peered":"2024-04-25T18:46:33.023970+0000","last_clean":"2024-04-25T18:46:33.023970+0000","last_became_active":"2024-04-25T18:42:28.887079+0000","last_became_peered":"2024-04-25T18:42:28.887079+0000","last_unstale":"2024-04-25T18:46:33.023970+0000","last_undegraded":"2024-04-25T18:46:33.023970+0000","last_fullsized":"2024-04-25T18:46:33.023970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:59:35.579036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"92'6","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892434+0000","last_change":"2024-04-25T18:42:24.846206+0000","last_active":"2024-04-25T18:46:32.892434+0000","last_peered":"2024-04-25T18:46:32.892434+0000","last_clean":"2024-04-25T18:46:32.892434+0000","last_became_active":"2024-04-25T18:42:24.845784+0000","last_became_peered":"2024-04-25T18:42:24.845784+0000","last_unstale":"2024-04-25T18:46:32.892434+0000","last_undegraded":"2024-04-25T18:46:32.892434+0000","last_fullsized":"2024-04-25T18:46:32.892434+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:23:45.837826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094919+0000","last_change":"2024-04-25T18:42:28.906361+0000","last_active":"2024-04-25T18:46:33.094919+0000","last_peered":"2024-04-25T18:46:33.094919+0000","last_clean":"2024-04-25T18:46:33.094919+0000","last_became_active":"2024-04-25T18:42:28.905955+0000","last_became_peered":"2024-04-25T18:42:28.905955+0000","last_unstale":"2024-04-25T18:46:33.094919+0000","last_undegraded":"2024-04-25T18:46:33.094919+0000","last_fullsized":"2024-04-25T18:46:33.094919+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:33:10.823229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936173+0000","last_change":"2024-04-25T18:42:24.837948+0000","last_active":"2024-04-25T18:46:32.936173+0000","last_peered":"2024-04-25T18:46:32.936173+0000","last_clean":"2024-04-25T18:46:32.936173+0000","last_became_active":"2024-04-25T18:42:24.837550+0000","last_became_peered":"2024-04-25T18:42:24.837550+0000","last_unstale":"2024-04-25T18:46:32.936173+0000","last_undegraded":"2024-04-25T18:46:32.936173+0000","last_fullsized":"2024-04-25T18:46:32.936173+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:24:52.483631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232141+0000","last_change":"2024-04-25T18:42:28.889021+0000","last_active":"2024-04-25T18:46:33.232141+0000","last_peered":"2024-04-25T18:46:33.232141+0000","last_clean":"2024-04-25T18:46:33.232141+0000","last_became_active":"2024-04-25T18:42:28.888653+0000","last_became_peered":"2024-04-25T18:42:28.888653+0000","last_unstale":"2024-04-25T18:46:33.232141+0000","last_undegraded":"2024-04-25T18:46:33.232141+0000","last_fullsized":"2024-04-25T18:46:33.232141+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:15.127944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"97'8","reported_seq":277,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:42.112917+0000","last_change":"2024-04-25T18:42:24.828830+0000","last_active":"2024-04-25T18:46:42.112917+0000","last_peered":"2024-04-25T18:46:42.112917+0000","last_clean":"2024-04-25T18:46:42.112917+0000","last_became_active":"2024-04-25T18:42:24.828438+0000","last_became_peered":"2024-04-25T18:42:24.828438+0000","last_unstale":"2024-04-25T18:46:42.112917+0000","last_undegraded":"2024-04-25T18:46:42.112917+0000","last_fullsized":"2024-04-25T18:46:42.112917+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:23.213745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2212,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":7,"num_write":7,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231813+0000","last_change":"2024-04-25T18:42:28.880322+0000","last_active":"2024-04-25T18:46:33.231813+0000","last_peered":"2024-04-25T18:46:33.231813+0000","last_clean":"2024-04-25T18:46:33.231813+0000","last_became_active":"2024-04-25T18:42:28.879975+0000","last_became_peered":"2024-04-25T18:42:28.879975+0000","last_unstale":"2024-04-25T18:46:33.231813+0000","last_undegraded":"2024-04-25T18:46:33.231813+0000","last_fullsized":"2024-04-25T18:46:33.231813+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:51:19.039619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"92'15","reported_seq":44,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422514+0000","last_change":"2024-04-25T18:42:55.144270+0000","last_active":"2024-04-25T18:46:33.422514+0000","last_peered":"2024-04-25T18:46:33.422514+0000","last_clean":"2024-04-25T18:46:33.422514+0000","last_became_active":"2024-04-25T18:42:55.113879+0000","last_became_peered":"2024-04-25T18:42:55.113879+0000","last_unstale":"2024-04-25T18:46:33.422514+0000","last_undegraded":"2024-04-25T18:46:33.422514+0000","last_fullsized":"2024-04-25T18:46:33.422514+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:57:11.722287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936373+0000","last_change":"2024-04-25T18:42:28.882759+0000","last_active":"2024-04-25T18:46:32.936373+0000","last_peered":"2024-04-25T18:46:32.936373+0000","last_clean":"2024-04-25T18:46:32.936373+0000","last_became_active":"2024-04-25T18:42:28.882367+0000","last_became_peered":"2024-04-25T18:42:28.882367+0000","last_unstale":"2024-04-25T18:46:32.936373+0000","last_undegraded":"2024-04-25T18:46:32.936373+0000","last_fullsized":"2024-04-25T18:46:32.936373+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:40.250143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"97'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024089+0000","last_change":"2024-04-25T18:42:24.833626+0000","last_active":"2024-04-25T18:46:33.024089+0000","last_peered":"2024-04-25T18:46:33.024089+0000","last_clean":"2024-04-25T18:46:33.024089+0000","last_became_active":"2024-04-25T18:42:24.833212+0000","last_became_peered":"2024-04-25T18:42:24.833212+0000","last_unstale":"2024-04-25T18:46:33.024089+0000","last_undegraded":"2024-04-25T18:46:33.024089+0000","last_fullsized":"2024-04-25T18:46:33.024089+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:37:54.439091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422673+0000","last_change":"2024-04-25T18:42:28.880904+0000","last_active":"2024-04-25T18:46:33.422673+0000","last_peered":"2024-04-25T18:46:33.422673+0000","last_clean":"2024-04-25T18:46:33.422673+0000","last_became_active":"2024-04-25T18:42:28.880486+0000","last_became_peered":"2024-04-25T18:42:28.880486+0000","last_unstale":"2024-04-25T18:46:33.422673+0000","last_undegraded":"2024-04-25T18:46:33.422673+0000","last_fullsized":"2024-04-25T18:46:33.422673+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:35:50.135977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889115+0000","last_change":"2024-04-25T18:42:24.848331+0000","last_active":"2024-04-25T18:46:32.889115+0000","last_peered":"2024-04-25T18:46:32.889115+0000","last_clean":"2024-04-25T18:46:32.889115+0000","last_became_active":"2024-04-25T18:42:24.847929+0000","last_became_peered":"2024-04-25T18:42:24.847929+0000","last_unstale":"2024-04-25T18:46:32.889115+0000","last_undegraded":"2024-04-25T18:46:32.889115+0000","last_fullsized":"2024-04-25T18:46:32.889115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:01.821175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933807+0000","last_change":"2024-04-25T18:42:28.897392+0000","last_active":"2024-04-25T18:46:32.933807+0000","last_peered":"2024-04-25T18:46:32.933807+0000","last_clean":"2024-04-25T18:46:32.933807+0000","last_became_active":"2024-04-25T18:42:28.896918+0000","last_became_peered":"2024-04-25T18:42:28.896918+0000","last_unstale":"2024-04-25T18:46:32.933807+0000","last_undegraded":"2024-04-25T18:46:32.933807+0000","last_fullsized":"2024-04-25T18:46:32.933807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:50.039803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"92'1","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232202+0000","last_change":"2024-04-25T18:42:24.838584+0000","last_active":"2024-04-25T18:46:33.232202+0000","last_peered":"2024-04-25T18:46:33.232202+0000","last_clean":"2024-04-25T18:46:33.232202+0000","last_became_active":"2024-04-25T18:42:24.838022+0000","last_became_peered":"2024-04-25T18:42:24.838022+0000","last_unstale":"2024-04-25T18:46:33.232202+0000","last_undegraded":"2024-04-25T18:46:33.232202+0000","last_fullsized":"2024-04-25T18:46:33.232202+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:53:33.171990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"96'7","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.933682+0000","last_change":"2024-04-25T18:42:28.894350+0000","last_active":"2024-04-25T18:46:32.933682+0000","last_peered":"2024-04-25T18:46:32.933682+0000","last_clean":"2024-04-25T18:46:32.933682+0000","last_became_active":"2024-04-25T18:42:28.893999+0000","last_became_peered":"2024-04-25T18:42:28.893999+0000","last_unstale":"2024-04-25T18:46:32.933682+0000","last_undegraded":"2024-04-25T18:46:32.933682+0000","last_fullsized":"2024-04-25T18:46:32.933682+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:49:12.959979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.888871+0000","last_change":"2024-04-25T18:43:55.423277+0000","last_active":"2024-04-25T18:46:32.888871+0000","last_peered":"2024-04-25T18:46:32.888871+0000","last_clean":"2024-04-25T18:46:32.888871+0000","last_became_active":"2024-04-25T18:43:55.421960+0000","last_became_peered":"2024-04-25T18:43:55.421960+0000","last_unstale":"2024-04-25T18:46:32.888871+0000","last_undegraded":"2024-04-25T18:46:32.888871+0000","last_fullsized":"2024-04-25T18:46:32.888871+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:47:27.858834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"96'6","reported_seq":270,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:42.112081+0000","last_change":"2024-04-25T18:42:24.849807+0000","last_active":"2024-04-25T18:46:42.112081+0000","last_peered":"2024-04-25T18:46:42.112081+0000","last_clean":"2024-04-25T18:46:42.112081+0000","last_became_active":"2024-04-25T18:42:24.849319+0000","last_became_peered":"2024-04-25T18:42:24.849319+0000","last_unstale":"2024-04-25T18:46:42.112081+0000","last_undegraded":"2024-04-25T18:46:42.112081+0000","last_fullsized":"2024-04-25T18:46:42.112081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:19:14.102832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":230,"num_read_kb":230,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024075+0000","last_change":"2024-04-25T18:42:28.902809+0000","last_active":"2024-04-25T18:46:33.024075+0000","last_peered":"2024-04-25T18:46:33.024075+0000","last_clean":"2024-04-25T18:46:33.024075+0000","last_became_active":"2024-04-25T18:42:28.899102+0000","last_became_peered":"2024-04-25T18:42:28.899102+0000","last_unstale":"2024-04-25T18:46:33.024075+0000","last_undegraded":"2024-04-25T18:46:33.024075+0000","last_fullsized":"2024-04-25T18:46:33.024075+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:22:18.367747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"92'3","reported_seq":39,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890279+0000","last_change":"2024-04-25T18:42:24.834546+0000","last_active":"2024-04-25T18:46:32.890279+0000","last_peered":"2024-04-25T18:46:32.890279+0000","last_clean":"2024-04-25T18:46:32.890279+0000","last_became_active":"2024-04-25T18:42:24.834047+0000","last_became_peered":"2024-04-25T18:42:24.834047+0000","last_unstale":"2024-04-25T18:46:32.890279+0000","last_undegraded":"2024-04-25T18:46:32.890279+0000","last_fullsized":"2024-04-25T18:46:32.890279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:53.362719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"97'148","reported_seq":413,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.036283+0000","last_change":"2024-04-25T18:40:46.774179+0000","last_active":"2024-04-25T18:46:35.036283+0000","last_peered":"2024-04-25T18:46:35.036283+0000","last_clean":"2024-04-25T18:46:35.036283+0000","last_became_active":"2024-04-25T18:40:46.773512+0000","last_became_peered":"2024-04-25T18:40:46.773512+0000","last_unstale":"2024-04-25T18:46:35.036283+0000","last_undegraded":"2024-04-25T18:46:35.036283+0000","last_fullsized":"2024-04-25T18:46:35.036283+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:36:47.863356+0000","last_clean_scrub_stamp":"2024-04-25T18:36:47.863356+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:07:34.190653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"96'5","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094794+0000","last_change":"2024-04-25T18:42:28.888887+0000","last_active":"2024-04-25T18:46:33.094794+0000","last_peered":"2024-04-25T18:46:33.094794+0000","last_clean":"2024-04-25T18:46:33.094794+0000","last_became_active":"2024-04-25T18:42:28.888519+0000","last_became_peered":"2024-04-25T18:42:28.888519+0000","last_unstale":"2024-04-25T18:46:33.094794+0000","last_undegraded":"2024-04-25T18:46:33.094794+0000","last_fullsized":"2024-04-25T18:46:33.094794+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:38:42.732815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232115+0000","last_change":"2024-04-25T18:42:24.833814+0000","last_active":"2024-04-25T18:46:33.232115+0000","last_peered":"2024-04-25T18:46:33.232115+0000","last_clean":"2024-04-25T18:46:33.232115+0000","last_became_active":"2024-04-25T18:42:24.833450+0000","last_became_peered":"2024-04-25T18:42:24.833450+0000","last_unstale":"2024-04-25T18:46:33.232115+0000","last_undegraded":"2024-04-25T18:46:33.232115+0000","last_fullsized":"2024-04-25T18:46:33.232115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:20.615731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"92'1","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024110+0000","last_change":"2024-04-25T18:42:24.850851+0000","last_active":"2024-04-25T18:46:33.024110+0000","last_peered":"2024-04-25T18:46:33.024110+0000","last_clean":"2024-04-25T18:46:33.024110+0000","last_became_active":"2024-04-25T18:42:24.850377+0000","last_became_peered":"2024-04-25T18:42:24.850377+0000","last_unstale":"2024-04-25T18:46:33.024110+0000","last_undegraded":"2024-04-25T18:46:33.024110+0000","last_fullsized":"2024-04-25T18:46:33.024110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:31.536061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"96'9","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095040+0000","last_change":"2024-04-25T18:42:28.897891+0000","last_active":"2024-04-25T18:46:33.095040+0000","last_peered":"2024-04-25T18:46:33.095040+0000","last_clean":"2024-04-25T18:46:33.095040+0000","last_became_active":"2024-04-25T18:42:28.896707+0000","last_became_peered":"2024-04-25T18:42:28.896707+0000","last_unstale":"2024-04-25T18:46:33.095040+0000","last_undegraded":"2024-04-25T18:46:33.095040+0000","last_fullsized":"2024-04-25T18:46:33.095040+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:03:21.530199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"97'100","reported_seq":226,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.726183+0000","last_change":"2024-04-25T18:42:24.851769+0000","last_active":"2024-04-25T18:46:35.726183+0000","last_peered":"2024-04-25T18:46:35.726183+0000","last_clean":"2024-04-25T18:46:35.726183+0000","last_became_active":"2024-04-25T18:42:24.850189+0000","last_became_peered":"2024-04-25T18:42:24.850189+0000","last_unstale":"2024-04-25T18:46:35.726183+0000","last_undegraded":"2024-04-25T18:46:35.726183+0000","last_fullsized":"2024-04-25T18:46:35.726183+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:47.498022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":184,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024773+0000","last_change":"2024-04-25T18:42:28.905142+0000","last_active":"2024-04-25T18:46:33.024773+0000","last_peered":"2024-04-25T18:46:33.024773+0000","last_clean":"2024-04-25T18:46:33.024773+0000","last_became_active":"2024-04-25T18:42:28.904771+0000","last_became_peered":"2024-04-25T18:42:28.904771+0000","last_unstale":"2024-04-25T18:46:33.024773+0000","last_undegraded":"2024-04-25T18:46:33.024773+0000","last_fullsized":"2024-04-25T18:46:33.024773+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:12.138365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"92'4","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.890406+0000","last_change":"2024-04-25T18:42:25.112074+0000","last_active":"2024-04-25T18:46:32.890406+0000","last_peered":"2024-04-25T18:46:32.890406+0000","last_clean":"2024-04-25T18:46:32.890406+0000","last_became_active":"2024-04-25T18:42:25.111583+0000","last_became_peered":"2024-04-25T18:42:25.111583+0000","last_unstale":"2024-04-25T18:46:32.890406+0000","last_undegraded":"2024-04-25T18:46:32.890406+0000","last_fullsized":"2024-04-25T18:46:32.890406+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:18.901321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424814+0000","last_change":"2024-04-25T18:42:28.912970+0000","last_active":"2024-04-25T18:46:33.424814+0000","last_peered":"2024-04-25T18:46:33.424814+0000","last_clean":"2024-04-25T18:46:33.424814+0000","last_became_active":"2024-04-25T18:42:28.912720+0000","last_became_peered":"2024-04-25T18:42:28.912720+0000","last_unstale":"2024-04-25T18:46:33.424814+0000","last_undegraded":"2024-04-25T18:46:33.424814+0000","last_fullsized":"2024-04-25T18:46:33.424814+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:54:06.690961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238097+0000","last_change":"2024-04-25T18:42:24.850583+0000","last_active":"2024-04-25T18:46:33.238097+0000","last_peered":"2024-04-25T18:46:33.238097+0000","last_clean":"2024-04-25T18:46:33.238097+0000","last_became_active":"2024-04-25T18:42:24.850234+0000","last_became_peered":"2024-04-25T18:42:24.850234+0000","last_unstale":"2024-04-25T18:46:33.238097+0000","last_undegraded":"2024-04-25T18:46:33.238097+0000","last_fullsized":"2024-04-25T18:46:33.238097+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:41:51.868753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023626+0000","last_change":"2024-04-25T18:42:28.886853+0000","last_active":"2024-04-25T18:46:33.023626+0000","last_peered":"2024-04-25T18:46:33.023626+0000","last_clean":"2024-04-25T18:46:33.023626+0000","last_became_active":"2024-04-25T18:42:28.886519+0000","last_became_peered":"2024-04-25T18:42:28.886519+0000","last_unstale":"2024-04-25T18:46:33.023626+0000","last_undegraded":"2024-04-25T18:46:33.023626+0000","last_fullsized":"2024-04-25T18:46:33.023626+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:05.306666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"92'3","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023651+0000","last_change":"2024-04-25T18:42:24.845534+0000","last_active":"2024-04-25T18:46:33.023651+0000","last_peered":"2024-04-25T18:46:33.023651+0000","last_clean":"2024-04-25T18:46:33.023651+0000","last_became_active":"2024-04-25T18:42:24.844854+0000","last_became_peered":"2024-04-25T18:42:24.844854+0000","last_unstale":"2024-04-25T18:46:33.023651+0000","last_undegraded":"2024-04-25T18:46:33.023651+0000","last_fullsized":"2024-04-25T18:46:33.023651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:00:02.707280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422592+0000","last_change":"2024-04-25T18:42:28.895130+0000","last_active":"2024-04-25T18:46:33.422592+0000","last_peered":"2024-04-25T18:46:33.422592+0000","last_clean":"2024-04-25T18:46:33.422592+0000","last_became_active":"2024-04-25T18:42:28.894721+0000","last_became_peered":"2024-04-25T18:42:28.894721+0000","last_unstale":"2024-04-25T18:46:33.422592+0000","last_undegraded":"2024-04-25T18:46:33.422592+0000","last_fullsized":"2024-04-25T18:46:33.422592+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:02:40.908868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"92'1","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892547+0000","last_change":"2024-04-25T18:42:24.846166+0000","last_active":"2024-04-25T18:46:32.892547+0000","last_peered":"2024-04-25T18:46:32.892547+0000","last_clean":"2024-04-25T18:46:32.892547+0000","last_became_active":"2024-04-25T18:42:24.845786+0000","last_became_peered":"2024-04-25T18:42:24.845786+0000","last_unstale":"2024-04-25T18:46:32.892547+0000","last_undegraded":"2024-04-25T18:46:32.892547+0000","last_fullsized":"2024-04-25T18:46:32.892547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:37:12.236589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"96'8","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024105+0000","last_change":"2024-04-25T18:42:28.910483+0000","last_active":"2024-04-25T18:46:33.024105+0000","last_peered":"2024-04-25T18:46:33.024105+0000","last_clean":"2024-04-25T18:46:33.024105+0000","last_became_active":"2024-04-25T18:42:28.910085+0000","last_became_peered":"2024-04-25T18:42:28.910085+0000","last_unstale":"2024-04-25T18:46:33.024105+0000","last_undegraded":"2024-04-25T18:46:33.024105+0000","last_fullsized":"2024-04-25T18:46:33.024105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:39.564381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"92'4","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.023735+0000","last_change":"2024-04-25T18:42:55.390330+0000","last_active":"2024-04-25T18:46:33.023735+0000","last_peered":"2024-04-25T18:46:33.023735+0000","last_clean":"2024-04-25T18:46:33.023735+0000","last_became_active":"2024-04-25T18:42:55.383893+0000","last_became_peered":"2024-04-25T18:42:55.383893+0000","last_unstale":"2024-04-25T18:46:33.023735+0000","last_undegraded":"2024-04-25T18:46:33.023735+0000","last_fullsized":"2024-04-25T18:46:33.023735+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:29:25.144279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,7],"acting":[8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"96'5","reported_seq":32,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.231981+0000","last_change":"2024-04-25T18:42:28.889391+0000","last_active":"2024-04-25T18:46:33.231981+0000","last_peered":"2024-04-25T18:46:33.231981+0000","last_clean":"2024-04-25T18:46:33.231981+0000","last_became_active":"2024-04-25T18:42:28.888894+0000","last_became_peered":"2024-04-25T18:42:28.888894+0000","last_unstale":"2024-04-25T18:46:33.231981+0000","last_undegraded":"2024-04-25T18:46:33.231981+0000","last_fullsized":"2024-04-25T18:46:33.231981+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:15:34.949526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"92'3","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892740+0000","last_change":"2024-04-25T18:42:25.111660+0000","last_active":"2024-04-25T18:46:32.892740+0000","last_peered":"2024-04-25T18:46:32.892740+0000","last_clean":"2024-04-25T18:46:32.892740+0000","last_became_active":"2024-04-25T18:42:25.111277+0000","last_became_peered":"2024-04-25T18:42:25.111277+0000","last_unstale":"2024-04-25T18:46:32.892740+0000","last_undegraded":"2024-04-25T18:46:32.892740+0000","last_fullsized":"2024-04-25T18:46:32.892740+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:42.491459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232147+0000","last_change":"2024-04-25T18:42:28.879829+0000","last_active":"2024-04-25T18:46:33.232147+0000","last_peered":"2024-04-25T18:46:33.232147+0000","last_clean":"2024-04-25T18:46:33.232147+0000","last_became_active":"2024-04-25T18:42:28.879453+0000","last_became_peered":"2024-04-25T18:42:28.879453+0000","last_unstale":"2024-04-25T18:46:33.232147+0000","last_undegraded":"2024-04-25T18:46:33.232147+0000","last_fullsized":"2024-04-25T18:46:33.232147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:50.782174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"97'65","reported_seq":129,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:35.205096+0000","last_change":"2024-04-25T18:42:24.837995+0000","last_active":"2024-04-25T18:46:35.205096+0000","last_peered":"2024-04-25T18:46:35.205096+0000","last_clean":"2024-04-25T18:46:35.205096+0000","last_became_active":"2024-04-25T18:42:24.837566+0000","last_became_peered":"2024-04-25T18:42:24.837566+0000","last_unstale":"2024-04-25T18:46:35.205096+0000","last_undegraded":"2024-04-25T18:46:35.205096+0000","last_fullsized":"2024-04-25T18:46:35.205096+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:06.899807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"96'7","reported_seq":34,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232014+0000","last_change":"2024-04-25T18:42:28.901983+0000","last_active":"2024-04-25T18:46:33.232014+0000","last_peered":"2024-04-25T18:46:33.232014+0000","last_clean":"2024-04-25T18:46:33.232014+0000","last_became_active":"2024-04-25T18:42:28.901661+0000","last_became_peered":"2024-04-25T18:42:28.901661+0000","last_unstale":"2024-04-25T18:46:33.232014+0000","last_undegraded":"2024-04-25T18:46:33.232014+0000","last_fullsized":"2024-04-25T18:46:33.232014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:32:13.555852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"92'2","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094761+0000","last_change":"2024-04-25T18:42:24.846457+0000","last_active":"2024-04-25T18:46:33.094761+0000","last_peered":"2024-04-25T18:46:33.094761+0000","last_clean":"2024-04-25T18:46:33.094761+0000","last_became_active":"2024-04-25T18:42:24.846081+0000","last_became_peered":"2024-04-25T18:42:24.846081+0000","last_unstale":"2024-04-25T18:46:33.094761+0000","last_undegraded":"2024-04-25T18:46:33.094761+0000","last_fullsized":"2024-04-25T18:46:33.094761+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:23.796232+0000","last_clean_scrub_stamp":"2024-04-25T18:42:23.796232+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:07.433857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"96'9","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.232252+0000","last_change":"2024-04-25T18:42:28.882069+0000","last_active":"2024-04-25T18:46:33.232252+0000","last_peered":"2024-04-25T18:46:33.232252+0000","last_clean":"2024-04-25T18:46:33.232252+0000","last_became_active":"2024-04-25T18:42:28.880578+0000","last_became_peered":"2024-04-25T18:42:28.880578+0000","last_unstale":"2024-04-25T18:46:33.232252+0000","last_undegraded":"2024-04-25T18:46:33.232252+0000","last_fullsized":"2024-04-25T18:46:33.232252+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:02:29.340426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"96'9","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422784+0000","last_change":"2024-04-25T18:42:28.880736+0000","last_active":"2024-04-25T18:46:33.422784+0000","last_peered":"2024-04-25T18:46:33.422784+0000","last_clean":"2024-04-25T18:46:33.422784+0000","last_became_active":"2024-04-25T18:42:28.880315+0000","last_became_peered":"2024-04-25T18:42:28.880315+0000","last_unstale":"2024-04-25T18:46:33.422784+0000","last_undegraded":"2024-04-25T18:46:33.422784+0000","last_fullsized":"2024-04-25T18:46:33.422784+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:25:09.036100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"96'12","reported_seq":40,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892470+0000","last_change":"2024-04-25T18:42:28.884752+0000","last_active":"2024-04-25T18:46:32.892470+0000","last_peered":"2024-04-25T18:46:32.892470+0000","last_clean":"2024-04-25T18:46:32.892470+0000","last_became_active":"2024-04-25T18:42:28.884297+0000","last_became_peered":"2024-04-25T18:42:28.884297+0000","last_unstale":"2024-04-25T18:46:32.892470+0000","last_undegraded":"2024-04-25T18:46:32.892470+0000","last_fullsized":"2024-04-25T18:46:32.892470+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:34.952969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"96'8","reported_seq":36,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892437+0000","last_change":"2024-04-25T18:42:28.906214+0000","last_active":"2024-04-25T18:46:32.892437+0000","last_peered":"2024-04-25T18:46:32.892437+0000","last_clean":"2024-04-25T18:46:32.892437+0000","last_became_active":"2024-04-25T18:42:28.905957+0000","last_became_peered":"2024-04-25T18:42:28.905957+0000","last_unstale":"2024-04-25T18:46:32.892437+0000","last_undegraded":"2024-04-25T18:46:32.892437+0000","last_fullsized":"2024-04-25T18:46:32.892437+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:56:55.895334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"96'7","reported_seq":35,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.424433+0000","last_change":"2024-04-25T18:42:28.905036+0000","last_active":"2024-04-25T18:46:33.424433+0000","last_peered":"2024-04-25T18:46:33.424433+0000","last_clean":"2024-04-25T18:46:33.424433+0000","last_became_active":"2024-04-25T18:42:28.904694+0000","last_became_peered":"2024-04-25T18:42:28.904694+0000","last_unstale":"2024-04-25T18:46:33.424433+0000","last_undegraded":"2024-04-25T18:46:33.424433+0000","last_fullsized":"2024-04-25T18:46:33.424433+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:25:58.678677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"96'4","reported_seq":31,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.094777+0000","last_change":"2024-04-25T18:42:28.891830+0000","last_active":"2024-04-25T18:46:33.094777+0000","last_peered":"2024-04-25T18:46:33.094777+0000","last_clean":"2024-04-25T18:46:33.094777+0000","last_became_active":"2024-04-25T18:42:28.891056+0000","last_became_peered":"2024-04-25T18:42:28.891056+0000","last_unstale":"2024-04-25T18:46:33.094777+0000","last_undegraded":"2024-04-25T18:46:33.094777+0000","last_fullsized":"2024-04-25T18:46:33.094777+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:22:00.558657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422751+0000","last_change":"2024-04-25T18:42:28.885655+0000","last_active":"2024-04-25T18:46:33.422751+0000","last_peered":"2024-04-25T18:46:33.422751+0000","last_clean":"2024-04-25T18:46:33.422751+0000","last_became_active":"2024-04-25T18:42:28.885334+0000","last_became_peered":"2024-04-25T18:42:28.885334+0000","last_unstale":"2024-04-25T18:46:33.422751+0000","last_undegraded":"2024-04-25T18:46:33.422751+0000","last_fullsized":"2024-04-25T18:46:33.422751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:33:08.603827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"96'15","reported_seq":42,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.892922+0000","last_change":"2024-04-25T18:42:28.881938+0000","last_active":"2024-04-25T18:46:32.892922+0000","last_peered":"2024-04-25T18:46:32.892922+0000","last_clean":"2024-04-25T18:46:32.892922+0000","last_became_active":"2024-04-25T18:42:28.880944+0000","last_became_peered":"2024-04-25T18:42:28.880944+0000","last_unstale":"2024-04-25T18:46:32.892922+0000","last_undegraded":"2024-04-25T18:46:32.892922+0000","last_fullsized":"2024-04-25T18:46:32.892922+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:03.229222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"96'2","reported_seq":29,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.024639+0000","last_change":"2024-04-25T18:42:28.906578+0000","last_active":"2024-04-25T18:46:33.024639+0000","last_peered":"2024-04-25T18:46:33.024639+0000","last_clean":"2024-04-25T18:46:33.024639+0000","last_became_active":"2024-04-25T18:42:28.905626+0000","last_became_peered":"2024-04-25T18:42:28.905626+0000","last_unstale":"2024-04-25T18:46:33.024639+0000","last_undegraded":"2024-04-25T18:46:33.024639+0000","last_fullsized":"2024-04-25T18:46:33.024639+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:34:22.770736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"96'6","reported_seq":33,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.936158+0000","last_change":"2024-04-25T18:42:28.897639+0000","last_active":"2024-04-25T18:46:32.936158+0000","last_peered":"2024-04-25T18:46:32.936158+0000","last_clean":"2024-04-25T18:46:32.936158+0000","last_became_active":"2024-04-25T18:42:28.897172+0000","last_became_peered":"2024-04-25T18:42:28.897172+0000","last_unstale":"2024-04-25T18:46:32.936158+0000","last_undegraded":"2024-04-25T18:46:32.936158+0000","last_fullsized":"2024-04-25T18:46:32.936158+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:20.976378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:32.889007+0000","last_change":"2024-04-25T18:43:55.436949+0000","last_active":"2024-04-25T18:46:32.889007+0000","last_peered":"2024-04-25T18:46:32.889007+0000","last_clean":"2024-04-25T18:46:32.889007+0000","last_became_active":"2024-04-25T18:43:55.435757+0000","last_became_peered":"2024-04-25T18:43:55.435757+0000","last_unstale":"2024-04-25T18:46:32.889007+0000","last_undegraded":"2024-04-25T18:46:32.889007+0000","last_fullsized":"2024-04-25T18:46:32.889007+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:46.736524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2a","version":"96'10","reported_seq":37,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.238289+0000","last_change":"2024-04-25T18:42:28.906133+0000","last_active":"2024-04-25T18:46:33.238289+0000","last_peered":"2024-04-25T18:46:33.238289+0000","last_clean":"2024-04-25T18:46:33.238289+0000","last_became_active":"2024-04-25T18:42:28.905759+0000","last_became_peered":"2024-04-25T18:42:28.905759+0000","last_unstale":"2024-04-25T18:46:33.238289+0000","last_undegraded":"2024-04-25T18:46:33.238289+0000","last_fullsized":"2024-04-25T18:46:33.238289+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:11:00.033489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"96'11","reported_seq":38,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.095622+0000","last_change":"2024-04-25T18:42:28.899521+0000","last_active":"2024-04-25T18:46:33.095622+0000","last_peered":"2024-04-25T18:46:33.095622+0000","last_clean":"2024-04-25T18:46:33.095622+0000","last_became_active":"2024-04-25T18:42:28.898820+0000","last_became_peered":"2024-04-25T18:42:28.898820+0000","last_unstale":"2024-04-25T18:46:33.095622+0000","last_undegraded":"2024-04-25T18:46:33.095622+0000","last_fullsized":"2024-04-25T18:46:33.095622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:33:41.478851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"96'8","reported_seq":41,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-25T18:46:33.422615+0000","last_change":"2024-04-25T18:42:54.683581+0000","last_active":"2024-04-25T18:46:33.422615+0000","last_peered":"2024-04-25T18:46:33.422615+0000","last_clean":"2024-04-25T18:46:33.422615+0000","last_became_active":"2024-04-25T18:42:54.683059+0000","last_became_peered":"2024-04-25T18:42:54.683059+0000","last_unstale":"2024-04-25T18:46:33.422615+0000","last_undegraded":"2024-04-25T18:46:33.422615+0000","last_fullsized":"2024-04-25T18:46:33.422615+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T18:42:27.829969+0000","last_clean_scrub_stamp":"2024-04-25T18:42:27.829969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:30:34.240498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":793940,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1756,"num_read_kb":1089,"num_write":420,"num_write_kb":2353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":694,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2617344,"data_stored":2381820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":84,"seq":360777252933,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1077248,"data_stored":523043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":76,"seq":326417514570,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":41552,"kb_used_data":2056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93703600,"statfs":{"total":95995035648,"available":95952486400,"internally_reserved":0,"allocated":2105344,"data_stored":1510744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":71,"seq":304942678096,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36780,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708372,"statfs":{"total":95995035648,"available":95957372928,"internally_reserved":0,"allocated":1478656,"data_stored":916383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":63,"seq":270582939735,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36768,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708384,"statfs":{"total":95995035648,"available":95957385216,"internally_reserved":0,"allocated":1466368,"data_stored":915609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":54,"seq":231928234075,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36976,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708176,"statfs":{"total":95995035648,"available":95957172224,"internally_reserved":0,"allocated":1679360,"data_stored":1117032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397601,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":522556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1146880,"data_stored":589407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738501,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1093632,"data_stored":523827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":590845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37228,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707924,"statfs":{"total":95995035648,"available":95956914176,"internally_reserved":0,"allocated":1814528,"data_stored":1244315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855467,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":524729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659240,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37432,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93707720,"statfs":{"total":95995035648,"available":95956705280,"internally_reserved":0,"allocated":2023424,"data_stored":1443324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":67058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":721732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":330490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":393260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":394034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":397910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T18:46:47.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:46:48.301 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:48 smithi114 ceph-mon[32988]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:48 smithi114 ceph-mon[32988]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:48.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:48 smithi063 ceph-mon[28006]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:48.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:48 smithi063 ceph-mon[28006]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:48 smithi070 ceph-mon[32820]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T18:46:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:48 smithi070 ceph-mon[32820]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 -- 172.21.15.63:0/4254758145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc06acc0 msgr2=0x7f95fc06b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 --2- 172.21.15.63:0/4254758145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc06acc0 0x7f95fc06b140 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009900 tx=0x7f95e402a410 comp rx=0 tx=0).stop 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 -- 172.21.15.63:0/4254758145 shutdown_connections 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 --2- 172.21.15.63:0/4254758145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc10e8a0 0x7f95fc110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 --2- 172.21.15.63:0/4254758145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc06acc0 0x7f95fc06b140 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 --2- 172.21.15.63:0/4254758145 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc06a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 -- 172.21.15.63:0/4254758145 >> 172.21.15.63:0/4254758145 conn(0x7f95fc076c50 msgr2=0x7f95fc077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 -- 172.21.15.63:0/4254758145 shutdown_connections 2024-04-25T18:46:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.091+0000 7f9600e00640 1 -- 172.21.15.63:0/4254758145 wait complete. 2024-04-25T18:46:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.092+0000 7f9600e00640 1 Processor -- start 2024-04-25T18:46:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 -- start start 2024-04-25T18:46:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc06acc0 0x7f95fc1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 0x7f95fc19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc114470 con 0x7f95fc10e8a0 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1142f0 con 0x7f95fc06acc0 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f9600e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1145f0 con 0x7f95fc102f00 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f95fa800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 0x7f95fc19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f95f9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc06acc0 0x7f95fc1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59378/0 (socket says 172.21.15.63:59378) 2024-04-25T18:46:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.093+0000 7f95f3e00640 1 -- 172.21.15.63:0/2130831234 learned_addr learned my addr 172.21.15.63:0/2130831234 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 -- 172.21.15.63:0/2130831234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 msgr2=0x7f95fc1a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 -- 172.21.15.63:0/2130831234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc06acc0 msgr2=0x7f95fc1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc06acc0 0x7f95fc1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 -- 172.21.15.63:0/2130831234 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc19b010 con 0x7f95fc10e8a0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95f3e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.094+0000 7f95fa800640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 0x7f95fc19a750 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f95f400ec50 tx=0x7f95f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.095+0000 7f95f2a00640 1 -- 172.21.15.63:0/2130831234 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4003b70 con 0x7f95fc10e8a0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.095+0000 7f95f2a00640 1 -- 172.21.15.63:0/2130831234 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95f4003d10 con 0x7f95fc10e8a0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.095+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95fc19b300 con 0x7f95fc10e8a0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.095+0000 7f95f2a00640 1 -- 172.21.15.63:0/2130831234 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4010830 con 0x7f95fc10e8a0 2024-04-25T18:46:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.095+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95fc1a7610 con 0x7f95fc10e8a0 2024-04-25T18:46:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.098+0000 7f95f2a00640 1 -- 172.21.15.63:0/2130831234 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f95f4003ec0 con 0x7f95fc10e8a0 2024-04-25T18:46:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.098+0000 7f9603dd9740 1 Processor -- start 2024-04-25T18:46:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 -- start start 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 0x563942ae44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ae4a00 0x563942ae6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563942a3f900 con 0x563942ad9820 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5639429e5ef0 con 0x563942ae4a00 2024-04-25T18:46:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f9603dd9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563942a3b8c0 con 0x563942ada140 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.099+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40916/0 (socket says 172.21.15.63:40916) 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 -- 172.21.15.63:0/4000294177 learned_addr learned my addr 172.21.15.63:0/4000294177 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f9e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 0x563942ae44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ae4a00 0x563942ae6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 -- 172.21.15.63:0/4000294177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 msgr2=0x563942ae44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 0x563942ae44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 -- 172.21.15.63:0/4000294177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ae4a00 msgr2=0x563942ae6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ae4a00 0x563942ae6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 -- 172.21.15.63:0/4000294177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563942752210 con 0x563942ad9820 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f9e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 0x563942ae44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f2a00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 0x7f95e00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f3e00640 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005e00 tx=0x7f95e4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef4eba84763b99d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.100+0000 7f95f2a00640 1 -- 172.21.15.63:0/2130831234 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f95f40d0f70 con 0x7f95fc10e8a0 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.101+0000 7f95f9e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 0x7f95e00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.101+0000 7f95eb400640 1 -- 172.21.15.63:0/4000294177 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4044070 con 0x563942ad9820 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.101+0000 7f95eb400640 1 -- 172.21.15.63:0/4000294177 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e4005020 con 0x563942ad9820 2024-04-25T18:46:49.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.101+0000 7f95eb400640 1 -- 172.21.15.63:0/4000294177 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4035040 con 0x563942ad9820 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- 172.21.15.63:0/4000294177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 msgr2=0x563942ad9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f95e4005e00 tx=0x7f95e4005d50 comp rx=0 tx=0).stop 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f95f9e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 0x7f95e00a6320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x563942adadc0 tx=0x7f95ec005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- 172.21.15.63:0/4000294177 shutdown_connections 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ae4a00 0x563942ae6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ada140 0x563942ae44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 --2- 172.21.15.63:0/4000294177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ad9820 0x563942ad9c00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- 172.21.15.63:0/4000294177 >> 172.21.15.63:0/4000294177 conn(0x563942ad5d30 msgr2=0x563942ad6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- 172.21.15.63:0/4000294177 shutdown_connections 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- 172.21.15.63:0/4000294177 wait complete. 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.102+0000 7f9603dd9740 1 -- start start 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 Processor -- start 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942ae1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563942ae75d0 con 0x563942ada140 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5639429f9310 con 0x563942ad9820 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563942ae7750 con 0x563942ae4a00 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f9603dd9740 10 client.0 _ll_register_callbacks cb 0x5639429c7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f95fa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f95f9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.103+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942ae1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.108 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58180/0 (socket says 172.21.15.63:58180) 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 -- 172.21.15.63:0/3000711381 learned_addr learned my addr 172.21.15.63:0/3000711381 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95fa800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59396/0 (socket says 172.21.15.63:59396) 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 -- 172.21.15.63:0/3000711381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 msgr2=0x563942ae23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 -- 172.21.15.63:0/3000711381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 msgr2=0x563942ae1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942ae1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 -- 172.21.15.63:0/3000711381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56394281e620 con 0x563942ad9820 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95fa800640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0045d0 tx=0x7f95ec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69677ba66c353d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.104+0000 7f95dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec01f640 con 0x563942ad9820 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95e4005020 con 0x563942ad9820 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f95ec01f7e0 con 0x563942ad9820 2024-04-25T18:46:49.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f95ec01f9e0 con 0x563942ad9820 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f9603dd9740 10 client.25169 fetch_fsmap learned FSMap version 24 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563942af38f0 con 0x563942ad9820 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95e4005020 con 0x563942ad9820 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.105+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f95ec027020 con 0x563942ad9820 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 10 client.25169 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 10 client.25169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 20 client.25169 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942af4db0 0x563942af7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 4 client.25169 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563942a538b0 con 0x563942af4db0 2024-04-25T18:46:49.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f95f3e00640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942af4db0 0x563942af7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f95f3e00640 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942af4db0 0x563942af7200 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c7bb59a49b14f80 server_cookie=a8dfa05abfb56412 in_seq=0 out_seq=0 2024-04-25T18:46:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.106+0000 7f95dfe00640 10 client.25169 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.107+0000 7f95dfe00640 1 -- 172.21.15.63:0/3000711381 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563942a538b0 con 0x563942af4db0 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.107+0000 7f95dfe00640 10 client.25169 handle_command_reply: tid=0 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.107+0000 7f9603dd9740 1 client.25169 shutdown 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 msgr2=0x563942ae1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0045d0 tx=0x7f95ec004730 comp rx=0 tx=0).stop 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 shutdown_connections 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942af4db0 0x563942af7200 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942ae23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942ae1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3000711381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942ae1930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 >> 172.21.15.63:0/3000711381 conn(0x563942ad5d30 msgr2=0x563942adbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f95dfe00640 0 client.25169 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 shutdown_connections 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 1 -- 172.21.15.63:0/3000711381 wait complete. 2024-04-25T18:46:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.108+0000 7f9603dd9740 20 client.25169 trim_cache size 0 max 16384 2024-04-25T18:46:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 Processor -- start 2024-04-25T18:46:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 -- start start 2024-04-25T18:46:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942af04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942afcb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1b24b0 con 0x563942ada140 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1afd80 con 0x563942ad9820 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f9603dd9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1adbd0 con 0x563942ae4a00 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40944/0 (socket says 172.21.15.63:40944) 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f95f3e00640 1 -- 172.21.15.63:0/3362037805 learned_addr learned my addr 172.21.15.63:0/3362037805 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942af04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.125+0000 7f95fa800640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942afcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 -- 172.21.15.63:0/3362037805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 msgr2=0x563942afcb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942afcb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 -- 172.21.15.63:0/3362037805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 msgr2=0x563942adc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.128 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 -- 172.21.15.63:0/3362037805 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563942ae0960 con 0x563942ad9820 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f3e00640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95f9e00640 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942af04c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f95ec029ce0 tx=0x7f95ec0092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b560c98cf51fadc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95eaa00640 1 -- 172.21.15.63:0/3362037805 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec035030 con 0x563942ad9820 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.126+0000 7f95eaa00640 1 -- 172.21.15.63:0/3362037805 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ec012d30 con 0x563942ad9820 2024-04-25T18:46:49.129 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f95eaa00640 1 -- 172.21.15.63:0/3362037805 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec01fe50 con 0x563942ad9820 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 -- 172.21.15.63:0/3362037805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 msgr2=0x563942af04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942af04c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f95ec029ce0 tx=0x7f95ec0092a0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 -- 172.21.15.63:0/3362037805 shutdown_connections 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ae4a00 0x563942afcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ada140 0x563942adc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 --2- 172.21.15.63:0/3362037805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ad9820 0x563942af04c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 -- 172.21.15.63:0/3362037805 >> 172.21.15.63:0/3362037805 conn(0x563942ad5d30 msgr2=0x563942ad4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 -- 172.21.15.63:0/3362037805 shutdown_connections 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.127+0000 7f9603dd9740 1 -- 172.21.15.63:0/3362037805 wait complete. 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 -- start start 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 Processor -- start 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ad9820 0x563942aece20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 0x563942aea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1ae240 con 0x563942ae4a00 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1af6b0 con 0x563942ada140 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc1ae910 con 0x563942ad9820 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.128+0000 7f9603dd9740 10 client.0 _ll_register_callbacks cb 0x7f95ec0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95fa800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 0x563942aea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ad9820 0x563942aece20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58188/0 (socket says 172.21.15.63:58188) 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 -- 172.21.15.63:0/1839289702 learned_addr learned my addr 172.21.15.63:0/1839289702 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 -- 172.21.15.63:0/1839289702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ad9820 msgr2=0x563942aece20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ad9820 0x563942aece20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 -- 172.21.15.63:0/1839289702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 msgr2=0x563942aea730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 0x563942aea730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 -- 172.21.15.63:0/1839289702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563942aeaf60 con 0x563942ada140 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95fa800640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 0x563942aea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95f3e00640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f95e4004650 tx=0x7f95e402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=152c2d13b9d69b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.129+0000 7f95dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4056060 con 0x563942ada140 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95ec012d30 con 0x563942ada140 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f95e4051b50 con 0x563942ada140 2024-04-25T18:46:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f95e4035040 con 0x563942ada140 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 10 client.25175 fetch_fsmap learned FSMap version 24 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563942b5b440 con 0x563942ada140 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95ec012d30 con 0x563942ada140 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f95e4051070 con 0x563942ada140 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 10 client.25175 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 10 client.25175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 20 client.25175 populate_metadata read hostname 'smithi063' 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942b5b6d0 0x563942b5dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 4 client.25175 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.130+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563942a4d320 con 0x563942b5b6d0 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.131+0000 7f95f9e00640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942b5b6d0 0x563942b5dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.131+0000 7f95f9e00640 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942b5b6d0 0x563942b5dac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c058faf1223cca2 server_cookie=f875fa1cb11af461 in_seq=0 out_seq=0 2024-04-25T18:46:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.131+0000 7f95dfe00640 10 client.25175 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f95dfe00640 1 -- 172.21.15.63:0/1839289702 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563942a4d320 con 0x563942b5b6d0 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f95dfe00640 10 client.25175 handle_command_reply: tid=0 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 client.25175 shutdown 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 msgr2=0x563942aea1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f95e4004650 tx=0x7f95e402a410 comp rx=0 tx=0).stop 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 shutdown_connections 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563942b5b6d0 0x563942b5dac0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563942ada140 0x563942aea1b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563942ae4a00 0x563942aea730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 --2- 172.21.15.63:0/1839289702 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563942ad9820 0x563942aece20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f95dfe00640 0 client.25175 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.145+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 >> 172.21.15.63:0/1839289702 conn(0x563942ad5d30 msgr2=0x563942ae18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.146+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 shutdown_connections 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.146+0000 7f9603dd9740 1 -- 172.21.15.63:0/1839289702 wait complete. 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.146+0000 7f9603dd9740 20 client.25175 trim_cache size 0 max 16384 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:46:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.146+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 msgr2=0x7f95e00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.146+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 0x7f95e00a6320 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x563942adadc0 tx=0x7f95ec005f90 comp rx=0 tx=0).stop 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 msgr2=0x7f95fc19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 0x7f95fc19a750 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f95f400ec50 tx=0x7f95f400c6a0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 shutdown_connections 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95e00a3e60 0x7f95e00a6320 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95fc10e8a0 0x7f95fc19a750 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95fc102f00 0x7f95fc1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 --2- 172.21.15.63:0/2130831234 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95fc06acc0 0x7f95fc1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 >> 172.21.15.63:0/2130831234 conn(0x7f95fc076c50 msgr2=0x7f95fc10f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 shutdown_connections 2024-04-25T18:46:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:46:49.147+0000 7f9600e00640 1 -- 172.21.15.63:0/2130831234 wait complete. 2024-04-25T18:46:50.565 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:50 smithi063 ceph-mon[28006]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:50.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:50 smithi114 ceph-mon[32988]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:50 smithi070 ceph-mon[32820]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 46 op/s 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 -- 172.21.15.63:0/3602003608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 msgr2=0x7f77cc102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 --2- 172.21.15.63:0/3602003608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc102df0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009990 tx=0x7f77b8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 -- 172.21.15.63:0/3602003608 shutdown_connections 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 --2- 172.21.15.63:0/3602003608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc103330 0x7f77cc10f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 --2- 172.21.15.63:0/3602003608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc108970 0x7f77cc108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 --2- 172.21.15.63:0/3602003608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc102df0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 -- 172.21.15.63:0/3602003608 >> 172.21.15.63:0/3602003608 conn(0x7f77cc0fe4c0 msgr2=0x7f77cc1008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 -- 172.21.15.63:0/3602003608 shutdown_connections 2024-04-25T18:46:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.888+0000 7f77cbe00700 1 -- 172.21.15.63:0/3602003608 wait complete. 2024-04-25T18:46:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 Processor -- start 2024-04-25T18:46:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 -- start start 2024-04-25T18:46:50.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:50.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 0x7f77cc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:50.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc108970 0x7f77cc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:50.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc112c00 con 0x7f77cc108970 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc112a80 con 0x7f77cc103330 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.890+0000 7f77cbe00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc112d80 con 0x7f77cc102970 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77caa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77caa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59436/0 (socket says 172.21.15.63:59436) 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77cb400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc108970 0x7f77cc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77caa00700 1 -- 172.21.15.63:0/840348060 learned_addr learned my addr 172.21.15.63:0/840348060 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 0x7f77cc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:50.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 -- 172.21.15.63:0/840348060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 msgr2=0x7f77cc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 -- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc108970 msgr2=0x7f77cc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc108970 0x7f77cc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.891+0000 7f77ca000700 1 -- 172.21.15.63:0/840348060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc1a75c0 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77ca000700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 0x7f77cc075280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f77b8000c00 tx=0x7f77b80047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8026070 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc1a7850 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77b8021410 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8017650 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.892+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc1a7d00 con 0x7f77cc103330 2024-04-25T18:46:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.893+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77cc069730 con 0x7f77cc103330 2024-04-25T18:46:50.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.895+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f77b8029070 con 0x7f77cc103330 2024-04-25T18:46:50.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.897+0000 7f77c3400700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 0x7f77b00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:50.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.897+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f77b80d16c0 con 0x7f77cc103330 2024-04-25T18:46:50.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.897+0000 7f77caa00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 0x7f77b00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:50.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.898+0000 7f77caa00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 0x7f77b00a6300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009ca0 tx=0x7f77bc009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:50.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:50.901+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77b805dff0 con 0x7f77cc103330 2024-04-25T18:46:51.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.179+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77cc10af20 con 0x7f77cc103330 2024-04-25T18:46:51.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.180+0000 7f77c3400700 1 -- 172.21.15.63:0/840348060 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11868 (secure 0 0 0) 0x7f77b8013070 con 0x7f77cc103330 2024-04-25T18:46:51.182 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:51.182 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:46:32.730913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:46:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 msgr2=0x7f77b00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 0x7f77b00a6300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009ca0 tx=0x7f77bc009260 comp rx=0 tx=0).stop 2024-04-25T18:46:51.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 msgr2=0x7f77cc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 0x7f77cc075280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f77b8000c00 tx=0x7f77b80047b0 comp rx=0 tx=0).stop 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 shutdown_connections 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77b00a3e40 0x7f77b00a6300 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77cc102970 0x7f77cc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77cc108970 0x7f77cc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 --2- 172.21.15.63:0/840348060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77cc103330 0x7f77cc075280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 >> 172.21.15.63:0/840348060 conn(0x7f77cc0fe4c0 msgr2=0x7f77cc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 shutdown_connections 2024-04-25T18:46:51.186 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:51.184+0000 7f77cbe00700 1 -- 172.21.15.63:0/840348060 wait complete. 2024-04-25T18:46:51.187 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-04-25T18:46:51.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:51 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/840348060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:51.593 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:51 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/840348060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:51.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:51 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/840348060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:52.008 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:46:52.495 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:52 smithi114 ceph-mon[32988]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-25T18:46:52.604 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:52 smithi070 ceph-mon[32820]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-25T18:46:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:52 smithi063 ceph-mon[28006]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-04-25T18:46:54.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:54 smithi063 ceph-mon[28006]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:54.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:54 smithi114 ceph-mon[32988]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:54 smithi070 ceph-mon[32820]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-25T18:46:54.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 -- 172.21.15.63:0/3866170199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06a750 msgr2=0x7fb2dc06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3866170199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06a750 0x7fb2dc06abb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009990 tx=0x7fb2cc009ca0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 -- 172.21.15.63:0/3866170199 shutdown_connections 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3866170199 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3866170199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3866170199 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06a750 0x7fb2dc06abb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 -- 172.21.15.63:0/3866170199 >> 172.21.15.63:0/3866170199 conn(0x7fb2dc0788f0 msgr2=0x7fb2dc078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 -- 172.21.15.63:0/3866170199 shutdown_connections 2024-04-25T18:46:54.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.973+0000 7fb2e3600700 1 -- 172.21.15.63:0/3866170199 wait complete. 2024-04-25T18:46:54.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.974+0000 7fb2e3600700 1 Processor -- start 2024-04-25T18:46:54.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.974+0000 7fb2e3600700 1 -- start start 2024-04-25T18:46:54.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:54.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:54.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc112c40 con 0x7fb2dc06a750 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc112ac0 con 0x7fb2dc06b0f0 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e3600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc112dc0 con 0x7fb2dc108a30 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e0c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40986/0 (socket says 172.21.15.63:40986) 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e0c00700 1 -- 172.21.15.63:0/3456191894 learned_addr learned my addr 172.21.15.63:0/3456191894 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2d3e00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:54.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e1600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.975+0000 7fb2e0c00700 1 -- 172.21.15.63:0/3456191894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 msgr2=0x7fb2dc19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e0c00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e0c00700 1 -- 172.21.15.63:0/3456191894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 msgr2=0x7fb2dc1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e0c00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e0c00700 1 -- 172.21.15.63:0/3456191894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc19e180 con 0x7fb2dc06a750 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e1600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2d3e00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e0c00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e9c0 tx=0x7fb2c400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400cbd0 con 0x7fb2dc06a750 2024-04-25T18:46:54.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc19e470 con 0x7fb2dc06a750 2024-04-25T18:46:54.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2c400cd70 con 0x7fb2dc06a750 2024-04-25T18:46:54.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4005610 con 0x7fb2dc06a750 2024-04-25T18:46:54.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.976+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc1a5cf0 con 0x7fb2dc06a750 2024-04-25T18:46:54.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.977+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2dc069730 con 0x7fb2dc06a750 2024-04-25T18:46:54.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.980+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7fb2c4004750 con 0x7fb2dc06a750 2024-04-25T18:46:54.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.981+0000 7fb2da000700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 0x7fb2bc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:54.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.981+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fb2c40d0eb0 con 0x7fb2dc06a750 2024-04-25T18:46:54.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.981+0000 7fb2d3e00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 0x7fb2bc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:54.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.982+0000 7fb2d3e00700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 0x7fb2bc0a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc078fd0 tx=0x7fb2cc01a040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:54.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:54.984+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2c4053ff0 con 0x7fb2dc06a750 2024-04-25T18:46:55.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.267+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2dc19a4d0 con 0x7fb2dc06a750 2024-04-25T18:46:55.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.269+0000 7fb2da000700 1 -- 172.21.15.63:0/3456191894 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11868 (secure 0 0 0) 0x7fb2c4087090 con 0x7fb2dc06a750 2024-04-25T18:46:55.271 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:55.271 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:46:32.730913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:46:55.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 msgr2=0x7fb2bc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:55.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 0x7fb2bc0a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc078fd0 tx=0x7fb2cc01a040 comp rx=0 tx=0).stop 2024-04-25T18:46:55.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 msgr2=0x7fb2dc199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:55.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e9c0 tx=0x7fb2c400ecd0 comp rx=0 tx=0).stop 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 shutdown_connections 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb2bc0a3e40 0x7fb2bc0a6300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb2dc108a30 0x7fb2dc19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2dc06a750 0x7fb2dc199170 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 --2- 172.21.15.63:0/3456191894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2dc06b0f0 0x7fb2dc1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 >> 172.21.15.63:0/3456191894 conn(0x7fb2dc0788f0 msgr2=0x7fb2dc0ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 shutdown_connections 2024-04-25T18:46:55.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:55.272+0000 7fb2e3600700 1 -- 172.21.15.63:0/3456191894 wait complete. 2024-04-25T18:46:55.275 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-04-25T18:46:55.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:55 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3456191894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:55.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:55 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3456191894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:55.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:55 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3456191894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:46:56.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T18:46:56.615 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:46:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:56 smithi063 ceph-mon[28006]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:46:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:56 smithi070 ceph-mon[32820]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:46:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:56 smithi114 ceph-mon[32988]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:46:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:58 smithi063 ceph-mon[28006]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:46:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:58 smithi070 ceph-mon[32820]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:46:58.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:58 smithi114 ceph-mon[32988]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:46:58.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 -- 172.21.15.63:0/3832244282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f0010f440 msgr2=0x7f4f00111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 --2- 172.21.15.63:0/3832244282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f0010f440 0x7f4f00111830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b0a0 tx=0x7f4ef800b3b0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 -- 172.21.15.63:0/3832244282 shutdown_connections 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 --2- 172.21.15.63:0/3832244282 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f00074c70 0x7f4f00075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 --2- 172.21.15.63:0/3832244282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f0010f440 0x7f4f00111830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 --2- 172.21.15.63:0/3832244282 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00075590 0x7f4f0010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 -- 172.21.15.63:0/3832244282 >> 172.21.15.63:0/3832244282 conn(0x7f4f00079ba0 msgr2=0x7f4f00079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 -- 172.21.15.63:0/3832244282 shutdown_connections 2024-04-25T18:46:58.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.959+0000 7f4f08000700 1 -- 172.21.15.63:0/3832244282 wait complete. 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.960+0000 7f4f08000700 1 Processor -- start 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.960+0000 7f4f08000700 1 -- start start 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00074c70 0x7f4f0019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f00075590 0x7f4f0019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4f00114e40 con 0x7f4f00075590 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f00114cc0 con 0x7f4f00074c70 2024-04-25T18:46:58.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f08000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4f00114fc0 con 0x7f4f0010f440 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f06000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f04c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f00075590 0x7f4f0019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f05600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00074c70 0x7f4f0019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f06000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49006/0 (socket says 172.21.15.63:49006) 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.961+0000 7f4f06000700 1 -- 172.21.15.63:0/351903450 learned_addr learned my addr 172.21.15.63:0/351903450 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 -- 172.21.15.63:0/351903450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00074c70 msgr2=0x7f4f0019b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00074c70 0x7f4f0019b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 -- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f00075590 msgr2=0x7f4f0019b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:58.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f00075590 0x7f4f0019b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 -- 172.21.15.63:0/351903450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f001a0370 con 0x7f4f0010f440 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f06000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b070 tx=0x7f4ef8004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8021070 con 0x7f4f0010f440 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ef8007e30 con 0x7f4f0010f440 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f001a0600 con 0x7f4f0010f440 2024-04-25T18:46:58.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.962+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80045e0 con 0x7f4f0010f440 2024-04-25T18:46:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.963+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f001a7ee0 con 0x7f4f0010f440 2024-04-25T18:46:58.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.964+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f00076360 con 0x7f4f0010f440 2024-04-25T18:46:58.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.966+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 24) ==== 138822+0+0 (secure 0 0 0) 0x7f4ef8003c90 con 0x7f4f0010f440 2024-04-25T18:46:58.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.967+0000 7f4efe000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 0x7f4ee80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:46:58.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.967+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f4ef80d1880 con 0x7f4f0010f440 2024-04-25T18:46:58.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.968+0000 7f4f05600700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 0x7f4ee80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:46:58.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.968+0000 7f4f05600700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 0x7f4ee80a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009c60 tx=0x7f4ef4009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:46:58.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:58.970+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ef809ddd0 con 0x7f4f0010f440 2024-04-25T18:46:59.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.201+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f0002df80 con 0x7f4f0010f440 2024-04-25T18:46:59.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.203+0000 7f4efe000700 1 -- 172.21.15.63:0/351903450 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+23114 (secure 0 0 0) 0x7f4ef809d6d0 con 0x7f4f0010f440 2024-04-25T18:46:59.205 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:46:59.205 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":97,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T18:46:32.881110+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3012688827":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/3560456479":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/2327478809":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6801/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/3611634789":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 msgr2=0x7f4ee80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 0x7f4ee80a6300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009c60 tx=0x7f4ef4009260 comp rx=0 tx=0).stop 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 msgr2=0x7f4f0019fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800b070 tx=0x7f4ef8004c60 comp rx=0 tx=0).stop 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 shutdown_connections 2024-04-25T18:46:59.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ee80a3e40 0x7f4ee80a6300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4f0010f440 0x7f4f0019fc30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f00074c70 0x7f4f0019b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 --2- 172.21.15.63:0/351903450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4f00075590 0x7f4f0019b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 >> 172.21.15.63:0/351903450 conn(0x7f4f00079ba0 msgr2=0x7f4f00100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 shutdown_connections 2024-04-25T18:46:59.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:46:59.207+0000 7f4f08000700 1 -- 172.21.15.63:0/351903450 wait complete. 2024-04-25T18:46:59.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:46:59 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/351903450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:46:59.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:46:59 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/351903450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:46:59.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:46:59 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/351903450' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T18:47:00.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-04-25T18:47:00.616 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-mon[28006]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:00 smithi070 ceph-mon[32820]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:00.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:00 smithi114 ceph-mon[32988]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:00.975 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:47:00.739+0000 7ff3d0a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:47:00.975 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:47:00.739+0000 7ff3d0a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:47:00.975 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:47:00.739+0000 7ff3d0a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:47:00.975 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:47:00.739+0000 7ff3d0a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:47:00.975 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 18:47:00 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T18:47:00.739+0000 7ff3d0a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T18:47:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 -- 172.21.15.63:0/832805504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444101500 msgr2=0x7f244410e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 --2- 172.21.15.63:0/832805504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444101500 0x7f244410e1c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2430009860 tx=0x7f243002a340 comp rx=0 tx=0).stop 2024-04-25T18:47:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 -- 172.21.15.63:0/832805504 shutdown_connections 2024-04-25T18:47:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 --2- 172.21.15.63:0/832805504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444101500 0x7f244410e1c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 --2- 172.21.15.63:0/832805504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 0x7f2444110af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 --2- 172.21.15.63:0/832805504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444100b50 0x7f2444100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 -- 172.21.15.63:0/832805504 >> 172.21.15.63:0/832805504 conn(0x7f24440fc7b0 msgr2=0x7f24440febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 -- 172.21.15.63:0/832805504 shutdown_connections 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.290+0000 7f244a800640 1 -- 172.21.15.63:0/832805504 wait complete. 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.291+0000 7f244a800640 1 Processor -- start 2024-04-25T18:47:02.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 -- start start 2024-04-25T18:47:02.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444100b50 0x7f24441a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 0x7f244419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2444114260 con 0x7f244410e700 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24441140e0 con 0x7f2444100b50 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f244a800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24441143e0 con 0x7f2444101500 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.292+0000 7f2449400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444100b50 0x7f24441a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2449e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 0x7f244419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49028/0 (socket says 172.21.15.63:49028) 2024-04-25T18:47:02.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 -- 172.21.15.63:0/3049666683 learned_addr learned my addr 172.21.15.63:0/3049666683 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 -- 172.21.15.63:0/3049666683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444100b50 msgr2=0x7f24441a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444100b50 0x7f24441a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 -- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 msgr2=0x7f244419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 0x7f244419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 -- 172.21.15.63:0/3049666683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244419af20 con 0x7f2444101500 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.293+0000 7f2448a00640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24300303f0 tx=0x7f2430032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.294+0000 7f2442000640 1 -- 172.21.15.63:0/3049666683 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430040020 con 0x7f2444101500 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.294+0000 7f2442000640 1 -- 172.21.15.63:0/3049666683 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2430030dd0 con 0x7f2444101500 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.294+0000 7f2442000640 1 -- 172.21.15.63:0/3049666683 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2430037040 con 0x7f2444101500 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.294+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244419b1b0 con 0x7f2444101500 2024-04-25T18:47:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.294+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24441a7340 con 0x7f2444101500 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.297+0000 7f2442000640 1 -- 172.21.15.63:0/3049666683 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f2430043020 con 0x7f2444101500 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.297+0000 7f244e922740 1 Processor -- start 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.297+0000 7f244e922740 1 -- start start 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 0x5644973d9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564497332900 con 0x5644973d7b90 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644972d8ef0 con 0x5644973cd2d0 2024-04-25T18:47:02.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f244e922740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56449732e8c0 con 0x5644973cc9b0 2024-04-25T18:47:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f2449400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 0x5644973d9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f2449400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46554/0 (socket says 172.21.15.63:46554) 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f2449400640 1 -- 172.21.15.63:0/720645320 learned_addr learned my addr 172.21.15.63:0/720645320 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.298+0000 7f2449e00640 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 -- 172.21.15.63:0/720645320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 msgr2=0x5644973ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 -- 172.21.15.63:0/720645320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 msgr2=0x5644973d9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 0x5644973d9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 -- 172.21.15.63:0/720645320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564497045210 con 0x5644973cd2d0 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2448a00640 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 0x5644973d9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2442000640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 0x7f242c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449400640 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2438009d20 tx=0x7f2438011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e208e391e3252be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2442000640 1 -- 172.21.15.63:0/3049666683 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f24300f05c0 con 0x7f2444101500 2024-04-25T18:47:02.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.299+0000 7f2449e00640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 0x7f242c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f2436600640 1 -- 172.21.15.63:0/720645320 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243801e070 con 0x5644973cd2d0 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f2436600640 1 -- 172.21.15.63:0/720645320 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2438019ae0 con 0x5644973cd2d0 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f2436600640 1 -- 172.21.15.63:0/720645320 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243800ea80 con 0x5644973cd2d0 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 -- 172.21.15.63:0/720645320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 msgr2=0x5644973d7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f2449e00640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 0x7f242c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f244406b470 tx=0x7f243c010040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2438009d20 tx=0x7f2438011810 comp rx=0 tx=0).stop 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 -- 172.21.15.63:0/720645320 shutdown_connections 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973cd2d0 0x5644973d7650 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973d7b90 0x5644973d9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 --2- 172.21.15.63:0/720645320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.300+0000 7f244e922740 1 -- 172.21.15.63:0/720645320 >> 172.21.15.63:0/720645320 conn(0x5644973c8ec0 msgr2=0x5644973c92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.301+0000 7f244e922740 1 -- 172.21.15.63:0/720645320 shutdown_connections 2024-04-25T18:47:02.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.301+0000 7f244e922740 1 -- 172.21.15.63:0/720645320 wait complete. 2024-04-25T18:47:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.301+0000 7f244e922740 1 -- start start 2024-04-25T18:47:02.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.301+0000 7f244e922740 1 Processor -- start 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973d50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973d55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5644973da7c0 con 0x5644973cd2d0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5644972ec310 con 0x5644973d7b90 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644973da940 con 0x5644973cc9b0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f244e922740 10 client.0 _ll_register_callbacks cb 0x5644972bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973d50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973d50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50860/0 (socket says 172.21.15.63:50860) 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f2448a00640 1 -- 172.21.15.63:0/76706791 learned_addr learned my addr 172.21.15.63:0/76706791 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f2449e00640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973d55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.302+0000 7f2449400640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 -- 172.21.15.63:0/76706791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 msgr2=0x5644973d4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973d4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 -- 172.21.15.63:0/76706791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 msgr2=0x5644973d50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973d50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 -- 172.21.15.63:0/76706791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564497111620 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f2449e00640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973d55e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f243c005fd0 tx=0x7f243c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d928188d4a2af0a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f242aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c020040 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f243c00dc80 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.303+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2438019ae0 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f243c032050 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 10 client.25211 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5644973e6ad0 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2438019ae0 con 0x5644973d7b90 2024-04-25T18:47:02.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f243c00b040 con 0x5644973d7b90 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 10 client.25211 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 10 client.25211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 20 client.25211 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5644973e7fe0 0x5644973ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 4 client.25211 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.304+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644973468b0 con 0x5644973e7fe0 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.305+0000 7f2449400640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5644973e7fe0 0x5644973ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.305+0000 7f2449400640 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5644973e7fe0 0x5644973ea3f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c298cb29f0244e97 server_cookie=44efdea5ac4659a6 in_seq=0 out_seq=0 2024-04-25T18:47:02.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.305+0000 7f242aa00640 10 client.25211 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f242aa00640 1 -- 172.21.15.63:0/76706791 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5644973468b0 con 0x5644973e7fe0 2024-04-25T18:47:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f242aa00640 10 client.25211 handle_command_reply: tid=0 2024-04-25T18:47:02.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 client.25211 shutdown 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 msgr2=0x5644973d55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973d55e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f243c005fd0 tx=0x7f243c004ab0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 shutdown_connections 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5644973e7fe0 0x5644973ea3f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.306+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973d55e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f242aa00640 0 client.25211 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973d50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 1 --2- 172.21.15.63:0/76706791 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 >> 172.21.15.63:0/76706791 conn(0x5644973c8ec0 msgr2=0x5644973cebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 shutdown_connections 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 1 -- 172.21.15.63:0/76706791 wait complete. 2024-04-25T18:47:02.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.307+0000 7f244e922740 20 client.25211 trim_cache size 0 max 16384 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 Processor -- start 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 -- start start 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973e3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24441b25c0 con 0x5644973cd2d0 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24441afe90 con 0x5644973d7b90 2024-04-25T18:47:02.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f244e922740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24441adce0 con 0x5644973cc9b0 2024-04-25T18:47:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.324+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46564/0 (socket says 172.21.15.63:46564) 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 -- 172.21.15.63:0/1610774794 learned_addr learned my addr 172.21.15.63:0/1610774794 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 -- 172.21.15.63:0/1610774794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 msgr2=0x5644973e3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973e3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 -- 172.21.15.63:0/1610774794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 msgr2=0x5644973cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 -- 172.21.15.63:0/1610774794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644973d30f0 con 0x5644973d7b90 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449400640 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973e3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.325+0000 7f2449e00640 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f243c003fd0 tx=0x7f243c009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153b59e0822633d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f2435c00640 1 -- 172.21.15.63:0/1610774794 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c020040 con 0x5644973d7b90 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f2435c00640 1 -- 172.21.15.63:0/1610774794 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c029070 con 0x5644973d7b90 2024-04-25T18:47:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f2435c00640 1 -- 172.21.15.63:0/1610774794 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c01a6b0 con 0x5644973d7b90 2024-04-25T18:47:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 -- 172.21.15.63:0/1610774794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 msgr2=0x5644973efd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f243c003fd0 tx=0x7f243c009f60 comp rx=0 tx=0).stop 2024-04-25T18:47:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 -- 172.21.15.63:0/1610774794 shutdown_connections 2024-04-25T18:47:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973efd00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 --2- 172.21.15.63:0/1610774794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973e3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 -- 172.21.15.63:0/1610774794 >> 172.21.15.63:0/1610774794 conn(0x5644973c8ec0 msgr2=0x5644973cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 -- 172.21.15.63:0/1610774794 shutdown_connections 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.326+0000 7f244e922740 1 -- 172.21.15.63:0/1610774794 wait complete. 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.327+0000 7f244e922740 1 -- start start 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.327+0000 7f244e922740 1 Processor -- start 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973dd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24441ae350 con 0x5644973cd2d0 2024-04-25T18:47:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24441af7c0 con 0x5644973d7b90 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f24441aea20 con 0x5644973cc9b0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f244e922740 10 client.0 _ll_register_callbacks cb 0x7f243c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50880/0 (socket says 172.21.15.63:50880) 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 -- 172.21.15.63:0/962938012 learned_addr learned my addr 172.21.15.63:0/962938012 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2449400640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2449e00640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973dd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 -- 172.21.15.63:0/962938012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 msgr2=0x5644973dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973dffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 -- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 msgr2=0x5644973dd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973dd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.328+0000 7f2448a00640 1 -- 172.21.15.63:0/962938012 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644973de170 con 0x5644973cd2d0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f2448a00640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f24300f3fd0 tx=0x7f243002a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e376764f5ede906 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f242aa00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24300bc180 con 0x5644973cd2d0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f24300c4d90 con 0x5644973cd2d0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f243c029070 con 0x5644973cd2d0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f24300bbd10 con 0x5644973cd2d0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f244e922740 10 client.15075 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56449744e660 con 0x5644973cd2d0 2024-04-25T18:47:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.329+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c029070 con 0x5644973cd2d0 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f24300f00e0 con 0x5644973cd2d0 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 10 client.15075 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 10 client.15075 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 20 client.15075 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56449744edb0 0x5644974511d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 4 client.15075 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5644973c8d10 con 0x56449744edb0 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.330+0000 7f2449400640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56449744edb0 0x5644974511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.331+0000 7f2449400640 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56449744edb0 0x5644974511d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b9e9517e22427fc server_cookie=86bf98fc7eadc440 in_seq=0 out_seq=0 2024-04-25T18:47:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.331+0000 7f242aa00640 10 client.15075 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f242aa00640 1 -- 172.21.15.63:0/962938012 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5644973c8d10 con 0x56449744edb0 2024-04-25T18:47:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f242aa00640 10 client.15075 handle_command_reply: tid=0 2024-04-25T18:47:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 client.15075 shutdown 2024-04-25T18:47:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 msgr2=0x5644973dd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f24300f3fd0 tx=0x7f243002a340 comp rx=0 tx=0).stop 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 shutdown_connections 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56449744edb0 0x5644974511d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5644973d7b90 0x5644973dd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5644973cd2d0 0x5644973dd390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 --2- 172.21.15.63:0/962938012 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5644973cc9b0 0x5644973dffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 >> 172.21.15.63:0/962938012 conn(0x5644973c8ec0 msgr2=0x5644973c7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f242aa00640 0 client.15075 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 shutdown_connections 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.340+0000 7f244e922740 1 -- 172.21.15.63:0/962938012 wait complete. 2024-04-25T18:47:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244e922740 20 client.15075 trim_cache size 0 max 16384 2024-04-25T18:47:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:47:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:47:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:47:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 msgr2=0x7f242c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 0x7f242c0a6370 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f244406b470 tx=0x7f243c010040 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 msgr2=0x7f24441a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24300303f0 tx=0x7f2430032da0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 shutdown_connections 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f242c0a3eb0 0x7f242c0a6370 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f244410e700 0x7f244419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2444101500 0x7f24441a0ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 --2- 172.21.15.63:0/3049666683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2444100b50 0x7f24441a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.341+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 >> 172.21.15.63:0/3049666683 conn(0x7f24440fc7b0 msgr2=0x7f24440feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.342+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 shutdown_connections 2024-04-25T18:47:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:02.342+0000 7f244a800640 1 -- 172.21.15.63:0/3049666683 wait complete. 2024-04-25T18:47:02.482 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:02 smithi063 ceph-mon[28006]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:47:02.482 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:02 smithi063 ceph-mon[28006]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-04-25T18:47:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:02 smithi070 ceph-mon[32820]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:47:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:02 smithi070 ceph-mon[32820]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-04-25T18:47:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:02 smithi114 ceph-mon[32988]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T18:47:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:02 smithi114 ceph-mon[32988]: mgrmap e25: x(active, since 28s), standbys: y, z 2024-04-25T18:47:03.741 INFO:teuthology.orchestra.run.smithi063.stdout:{"run_dir":"/var/run/ceph"} 2024-04-25T18:47:03.742 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-04-25T18:47:03.742 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with random seed 621586 2024-04-25T18:47:03.742 DEBUG:tasks.quiescer:Yielding 2024-04-25T18:47:03.743 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T18:47:03.756 INFO:tasks.workunit:Pulling workunits from ref 7797745e6d8b3ac036504dc61491d368615024da 2024-04-25T18:47:03.757 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T18:47:03.757 INFO:tasks.workunit:timeout=3h 2024-04-25T18:47:03.757 INFO:tasks.workunit:cleanup=True 2024-04-25T18:47:03.757 DEBUG:teuthology.orchestra.run.smithi063:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T18:47:03.758 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:47:03.815 INFO:teuthology.orchestra.run.smithi063.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T18:47:03.815 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T18:47:03.815 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-25T18:47:03.815 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T18:47:03.815 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T18:47:03.816 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 18:43:13.076708692 +0000 2024-04-25T18:47:03.816 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 18:43:13.076708692 +0000 2024-04-25T18:47:03.816 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 18:46:31.674469507 +0000 2024-04-25T18:47:03.816 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 18:43:13.076708692 +0000 2024-04-25T18:47:03.816 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T18:47:03.816 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T18:47:03.949 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7797745e6d8b3ac036504dc61491d368615024da 2024-04-25T18:47:04.008 INFO:tasks.workunit.client.0.smithi063.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T18:47:04.218 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:04.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:04 smithi063 ceph-mon[28006]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:04 smithi070 ceph-mon[32820]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:04 smithi114 ceph-mon[32988]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 9 op/s 2024-04-25T18:47:06.454 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:06 smithi063 ceph-mon[28006]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-25T18:47:06.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.521+0000 7fd9ea800700 1 -- 172.21.15.63:0/431384615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4103330 msgr2=0x7fd9e410f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:06.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.521+0000 7fd9ea800700 1 --2- 172.21.15.63:0/431384615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4103330 0x7fd9e410f840 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00b0a0 tx=0x7fd9dc00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 -- 172.21.15.63:0/431384615 shutdown_connections 2024-04-25T18:47:06.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 --2- 172.21.15.63:0/431384615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4103330 0x7fd9e410f840 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 --2- 172.21.15.63:0/431384615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 0x7fd9e4108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 --2- 172.21.15.63:0/431384615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4102970 0x7fd9e4102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 -- 172.21.15.63:0/431384615 >> 172.21.15.63:0/431384615 conn(0x7fd9e40fe4c0 msgr2=0x7fd9e41008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:06.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 -- 172.21.15.63:0/431384615 shutdown_connections 2024-04-25T18:47:06.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.522+0000 7fd9ea800700 1 -- 172.21.15.63:0/431384615 wait complete. 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 Processor -- start 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 -- start start 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 0x7fd9e419b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 0x7fd9e419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4112b50 con 0x7fd9e4108970 2024-04-25T18:47:06.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e41129d0 con 0x7fd9e4103330 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.523+0000 7fd9ea800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4112cd0 con 0x7fd9e4102970 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 0x7fd9e419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46598/0 (socket says 172.21.15.63:46598) 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 -- 172.21.15.63:0/395536801 learned_addr learned my addr 172.21.15.63:0/395536801 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e9400700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 0x7fd9e419b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 -- 172.21.15.63:0/395536801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 msgr2=0x7fd9e419b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 0x7fd9e419b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 -- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 msgr2=0x7fd9e419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 0x7fd9e419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e8a00700 1 -- 172.21.15.63:0/395536801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e41a0150 con 0x7fd9e4103330 2024-04-25T18:47:06.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.524+0000 7fd9e9400700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 0x7fd9e419b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9e8a00700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000e9a0 tx=0x7fd9d000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d000ca10 con 0x7fd9e4103330 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e41a0440 con 0x7fd9e4103330 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d000cbb0 con 0x7fd9e4103330 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0018680 con 0x7fd9e4103330 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.525+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e41a7cc0 con 0x7fd9e4103330 2024-04-25T18:47:06.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.526+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e4069730 con 0x7fd9e4103330 2024-04-25T18:47:06.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.529+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fd9d0010430 con 0x7fd9e4103330 2024-04-25T18:47:06.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.530+0000 7fd9e2000700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 0x7fd9cc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:06.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.530+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fd9d00d0c90 con 0x7fd9e4103330 2024-04-25T18:47:06.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.530+0000 7fd9e9400700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 0x7fd9cc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:06.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.531+0000 7fd9e9400700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 0x7fd9cc0a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800afa0 tx=0x7fd9d800a380 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:06.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.534+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9d009d070 con 0x7fd9e4103330 2024-04-25T18:47:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:06 smithi070 ceph-mon[32820]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-25T18:47:06.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:06 smithi114 ceph-mon[32988]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-04-25T18:47:06.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.834+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9e41a7e90 con 0x7fd9e4103330 2024-04-25T18:47:06.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.835+0000 7fd9e2000700 1 -- 172.21.15.63:0/395536801 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11868 (secure 0 0 0) 0x7fd9d009c970 con 0x7fd9e4103330 2024-04-25T18:47:06.837 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:47:06.837 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:46:32.730913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:47:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 msgr2=0x7fd9cc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 0x7fd9cc0a6390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800afa0 tx=0x7fd9d800a380 comp rx=0 tx=0).stop 2024-04-25T18:47:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 msgr2=0x7fd9e419b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000e9a0 tx=0x7fd9d000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 shutdown_connections 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd9cc0a3ed0 0x7fd9cc0a6390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd9e4102970 0x7fd9e419b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9e4108970 0x7fd9e419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 --2- 172.21.15.63:0/395536801 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9e4103330 0x7fd9e419b680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 >> 172.21.15.63:0/395536801 conn(0x7fd9e40fe4c0 msgr2=0x7fd9e410e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 shutdown_connections 2024-04-25T18:47:06.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:06.838+0000 7fd9ea800700 1 -- 172.21.15.63:0/395536801 wait complete. 2024-04-25T18:47:06.841 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-04-25T18:47:07.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:07 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/395536801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:07.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:07 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/395536801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:07.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T18:42:31.925526+0000', 'modified': '2024-04-25T18:46:32.730913+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24533, 'mds_1': 24520, 'mds_2': 24508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/3182056159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 3182056159}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 3182056159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6837/4043589496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 4043589496}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 4043589496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24496': {'gid': 24496, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.114:6835/3172988525', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6834', 'nonce': 3172988525}, {'type': 'v1', 'addr': '172.21.15.114:6835', 'nonce': 3172988525}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.114:6837/1526734616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6836', 'nonce': 1526734616}, {'type': 'v1', 'addr': '172.21.15.114:6837', 'nonce': 1526734616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24520': {'gid': 24520, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.114:6839/2417088150', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.114:6838', 'nonce': 2417088150}, {'type': 'v1', 'addr': '172.21.15.114:6839', 'nonce': 2417088150}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.70:6839/1944035005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 1944035005}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 1944035005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24533, 'qdb_cluster': [24520, 24533, 24508]} 2024-04-25T18:47:07.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-25T18:47:07.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.c status 2024-04-25T18:47:07.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:07 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/395536801' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:08.271 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:08.667 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:08 smithi063 ceph-mon[28006]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:08 smithi070 ceph-mon[32820]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:08 smithi114 ceph-mon[32988]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:10.480 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:10 smithi063 ceph-mon[28006]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:10 smithi070 ceph-mon[32820]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:10 smithi114 ceph-mon[32988]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 -- 172.21.15.63:0/267893398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 msgr2=0x7fa288111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 --2- 172.21.15.63:0/267893398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa288111830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa280009f90 tx=0x7fa28000b330 comp rx=0 tx=0).stop 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 -- 172.21.15.63:0/267893398 shutdown_connections 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 --2- 172.21.15.63:0/267893398 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa288111830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 --2- 172.21.15.63:0/267893398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa288075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 --2- 172.21.15.63:0/267893398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28810ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 -- 172.21.15.63:0/267893398 >> 172.21.15.63:0/267893398 conn(0x7fa288079ba0 msgr2=0x7fa288079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.836+0000 7fa28e800700 1 -- 172.21.15.63:0/267893398 shutdown_connections 2024-04-25T18:47:10.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.837+0000 7fa28e800700 1 -- 172.21.15.63:0/267893398 wait complete. 2024-04-25T18:47:10.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.838+0000 7fa28e800700 1 Processor -- start 2024-04-25T18:47:10.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.838+0000 7fa28e800700 1 -- start start 2024-04-25T18:47:10.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.838+0000 7fa28e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.838+0000 7fa28e800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa28e800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa28819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa28e800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa288114e40 con 0x7fa288075590 2024-04-25T18:47:10.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa28e800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa288114cc0 con 0x7fa288074c70 2024-04-25T18:47:10.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa28e800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa288114fc0 con 0x7fa28810f440 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44526/0 (socket says 172.21.15.63:44526) 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 -- 172.21.15.63:0/87298448 learned_addr learned my addr 172.21.15.63:0/87298448 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287e00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa28819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa286a00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 -- 172.21.15.63:0/87298448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 msgr2=0x7fa28819fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa28819fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 -- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 msgr2=0x7fa28819b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28819b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa287400700 1 -- 172.21.15.63:0/87298448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2881a0370 con 0x7fa288074c70 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.839+0000 7fa286a00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:10.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa287400700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa27800e9c0 tx=0x7fa27800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa27fe00700 1 -- 172.21.15.63:0/87298448 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27800cbd0 con 0x7fa288074c70 2024-04-25T18:47:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2881a0660 con 0x7fa288074c70 2024-04-25T18:47:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa27fe00700 1 -- 172.21.15.63:0/87298448 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27800cd70 con 0x7fa288074c70 2024-04-25T18:47:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa27fe00700 1 -- 172.21.15.63:0/87298448 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278005610 con 0x7fa288074c70 2024-04-25T18:47:10.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.840+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2881a7ee0 con 0x7fa288074c70 2024-04-25T18:47:10.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.844+0000 7fa27fe00700 1 -- 172.21.15.63:0/87298448 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa278004750 con 0x7fa288074c70 2024-04-25T18:47:10.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.846+0000 7fa27fe00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 0x7fa26c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.847+0000 7fa27fe00700 1 -- 172.21.15.63:0/87298448 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa2780d0e60 con 0x7fa288074c70 2024-04-25T18:47:10.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.847+0000 7fa286a00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 0x7fa26c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.847+0000 7fa29916eb80 1 Processor -- start 2024-04-25T18:47:10.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 -- start start 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa286a00700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 0x7fa26c0a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa288111140 tx=0x7fa274009310 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b40420 0x55fce6b4ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b4b2d0 0x55fce6b4d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fce6a8ae40 con 0x55fce6b40420 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fce6a316e0 con 0x55fce6b4b2d0 2024-04-25T18:47:10.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa29916eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fce6a86e00 con 0x55fce6b3fb00 2024-04-25T18:47:10.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa287400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa287400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47432/0 (socket says 172.21.15.63:47432) 2024-04-25T18:47:10.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa286a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b40420 0x55fce6b4ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.848+0000 7fa287400700 1 -- 172.21.15.63:0/50719672 learned_addr learned my addr 172.21.15.63:0/50719672 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:10.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287e00700 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b4b2d0 0x55fce6b4d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 -- 172.21.15.63:0/50719672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b4b2d0 msgr2=0x55fce6b4d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b4b2d0 0x55fce6b4d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 -- 172.21.15.63:0/50719672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b40420 msgr2=0x55fce6b4ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b40420 0x55fce6b4ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 -- 172.21.15.63:0/50719672 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fce6a27630 con 0x55fce6b3fb00 2024-04-25T18:47:10.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa287400700 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa27809c840 tx=0x7fa27809c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b330cd95c56571b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.849+0000 7fa27cc00700 1 -- 172.21.15.63:0/50719672 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780b6070 con 0x55fce6b3fb00 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.850+0000 7fa27cc00700 1 -- 172.21.15.63:0/50719672 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27809cb40 con 0x55fce6b3fb00 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.850+0000 7fa27cc00700 1 -- 172.21.15.63:0/50719672 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27809cda0 con 0x55fce6b3fb00 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.850+0000 7fa29916eb80 1 -- 172.21.15.63:0/50719672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 msgr2=0x55fce6b3fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.850+0000 7fa29916eb80 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa27809c840 tx=0x7fa27809c750 comp rx=0 tx=0).stop 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 -- 172.21.15.63:0/50719672 shutdown_connections 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b3fee0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b4b2d0 0x55fce6b4d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 --2- 172.21.15.63:0/50719672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b40420 0x55fce6b4ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 -- 172.21.15.63:0/50719672 >> 172.21.15.63:0/50719672 conn(0x55fce6a66550 msgr2=0x55fce6b33740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 -- 172.21.15.63:0/50719672 shutdown_connections 2024-04-25T18:47:10.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 -- 172.21.15.63:0/50719672 wait complete. 2024-04-25T18:47:10.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.851+0000 7fa29916eb80 1 -- start start 2024-04-25T18:47:10.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 Processor -- start 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b4aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b47d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b48280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b4de50 con 0x55fce6b4b2d0 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fce6a44810 con 0x55fce6b3fb00 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b4dfd0 con 0x55fce6b40420 2024-04-25T18:47:10.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.852+0000 7fa29916eb80 10 client.0 _ll_register_callbacks cb 0x55fce6a236d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b48280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b48280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40304/0 (socket says 172.21.15.63:40304) 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287e00700 1 -- 172.21.15.63:0/4015912897 learned_addr learned my addr 172.21.15.63:0/4015912897 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b4aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa286a00700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b47d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 -- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 msgr2=0x55fce6b47d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b47d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 -- 172.21.15.63:0/4015912897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 msgr2=0x55fce6b48280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b48280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.853+0000 7fa287400700 1 -- 172.21.15.63:0/4015912897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fce69d8880 con 0x55fce6b3fb00 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa287400700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b4aa40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa2780a10b0 tx=0x7fa278098ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98712834784ae496 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa271600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780b6040 con 0x55fce6b3fb00 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa2780a5320 con 0x55fce6b3fb00 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa27809cb40 con 0x55fce6b3fb00 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa2780a0ef0 con 0x55fce6b3fb00 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa29916eb80 10 client.25235 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:10.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fce6a21aa0 con 0x55fce6b3fb00 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.854+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27809cb40 con 0x55fce6b3fb00 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa278067310 con 0x55fce6b3fb00 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 10 client.25235 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 10 client.25235 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 20 client.25235 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6b59840 0x55fce6b5b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 4 client.25235 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:10.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fce6a98630 con 0x55fce6b59840 2024-04-25T18:47:10.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.855+0000 7fa286a00700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6b59840 0x55fce6b5b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.856+0000 7fa286a00700 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6b59840 0x55fce6b5b970 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54ae540daf68d39 server_cookie=be4c2d60fd40916d in_seq=0 out_seq=0 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.856+0000 7fa271600700 10 client.25235 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa271600700 1 -- 172.21.15.63:0/4015912897 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fce6a98630 con 0x55fce6b59840 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa271600700 10 client.25235 handle_command_reply: tid=0 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 client.25235 shutdown 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 msgr2=0x55fce6b4aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b4aa40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa2780a10b0 tx=0x7fa278098ee0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 shutdown_connections 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6b59840 0x55fce6b5b970 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b48280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b4aa40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa271600700 0 client.25235 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 --2- 172.21.15.63:0/4015912897 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b47d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.857+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 >> 172.21.15.63:0/4015912897 conn(0x55fce6a66550 msgr2=0x55fce6b32950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.858+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 shutdown_connections 2024-04-25T18:47:10.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.858+0000 7fa29916eb80 1 -- 172.21.15.63:0/4015912897 wait complete. 2024-04-25T18:47:10.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.858+0000 7fa29916eb80 20 client.25235 trim_cache size 0 max 16384 2024-04-25T18:47:10.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 Processor -- start 2024-04-25T18:47:10.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 -- start start 2024-04-25T18:47:10.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b43440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b64dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b59cb0 con 0x55fce6b4b2d0 2024-04-25T18:47:10.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b5a380 con 0x55fce6b3fb00 2024-04-25T18:47:10.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.881+0000 7fa29916eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b5aa50 con 0x55fce6b40420 2024-04-25T18:47:10.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa286a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b43440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44546/0 (socket says 172.21.15.63:44546) 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 -- 172.21.15.63:0/2958395852 learned_addr learned my addr 172.21.15.63:0/2958395852 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287e00700 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b64dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 -- 172.21.15.63:0/2958395852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 msgr2=0x55fce6b43440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b43440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 -- 172.21.15.63:0/2958395852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 msgr2=0x55fce6b64dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b64dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 -- 172.21.15.63:0/2958395852 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fce6b5a870 con 0x55fce6b3fb00 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa286a00700 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.882+0000 7fa287400700 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa278098f50 tx=0x7fa278098e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7580661eab9bdc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.883+0000 7fa273e00700 1 -- 172.21.15.63:0/2958395852 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780b6040 con 0x55fce6b3fb00 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.883+0000 7fa273e00700 1 -- 172.21.15.63:0/2958395852 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27809cb40 con 0x55fce6b3fb00 2024-04-25T18:47:10.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.883+0000 7fa273e00700 1 -- 172.21.15.63:0/2958395852 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27809c4a0 con 0x55fce6b3fb00 2024-04-25T18:47:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 -- 172.21.15.63:0/2958395852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 msgr2=0x55fce6b59e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa278098f50 tx=0x7fa278098e70 comp rx=0 tx=0).stop 2024-04-25T18:47:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 -- 172.21.15.63:0/2958395852 shutdown_connections 2024-04-25T18:47:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b64dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b3fb00 0x55fce6b59e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 --2- 172.21.15.63:0/2958395852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b40420 0x55fce6b43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 -- 172.21.15.63:0/2958395852 >> 172.21.15.63:0/2958395852 conn(0x55fce6a66550 msgr2=0x55fce6b42d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 -- 172.21.15.63:0/2958395852 shutdown_connections 2024-04-25T18:47:10.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.884+0000 7fa29916eb80 1 -- 172.21.15.63:0/2958395852 wait complete. 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.885+0000 7fa29916eb80 1 -- start start 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.885+0000 7fa29916eb80 1 Processor -- start 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.885+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 0x55fce6b65110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b5ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b5b7f0 con 0x55fce6b4b2d0 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b5b120 con 0x55fce6b40420 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fce6b5bec0 con 0x55fce6b3fb00 2024-04-25T18:47:10.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa29916eb80 10 client.0 _ll_register_callbacks cb 0x55fce6a236d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:10.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa287400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa287400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47454/0 (socket says 172.21.15.63:47454) 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa287400700 1 -- 172.21.15.63:0/3573614374 learned_addr learned my addr 172.21.15.63:0/3573614374 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 0x55fce6b65110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa287e00700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b5ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 -- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 msgr2=0x55fce6b64bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 -- 172.21.15.63:0/3573614374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 msgr2=0x55fce6b5ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b5ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa286a00700 1 -- 172.21.15.63:0/3573614374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fce6b5f750 con 0x55fce6b40420 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.886+0000 7fa287400700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa286a00700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 0x55fce6b65110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55fce6b34bb0 tx=0x7fa274004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58fc4f2b557386f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa271600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa274012650 con 0x55fce6b40420 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa27809cb40 con 0x55fce6b40420 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa2740127f0 con 0x55fce6b40420 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa2740129f0 con 0x55fce6b40420 2024-04-25T18:47:10.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa29916eb80 10 client.25241 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:10.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.887+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fce6bc20b0 con 0x55fce6b40420 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa27809cb40 con 0x55fce6b40420 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa274033070 con 0x55fce6b40420 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 10 client.25241 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 10 client.25241 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 20 client.25241 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6bc6c90 0x55fce6bc59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:10.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 4 client.25241 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.888+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "status"}) -- 0x55fce6a926d0 con 0x55fce6bc6c90 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa287400700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6bc6c90 0x55fce6bc59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa287400700 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6bc6c90 0x55fce6bc59a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d4dc8c9c63dd54a server_cookie=200f8a8ab8638fb8 in_seq=0 out_seq=0 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa271600700 10 client.25241 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa271600700 1 -- 172.21.15.63:0/3573614374 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fce6a926d0 con 0x55fce6bc6c90 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa271600700 10 client.25241 handle_command_reply: tid=0 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.889+0000 7fa29916eb80 1 client.25241 shutdown 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 msgr2=0x55fce6b65110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 0x55fce6b65110 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55fce6b34bb0 tx=0x7fa274004750 comp rx=0 tx=0).stop 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 shutdown_connections 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55fce6bc6c90 0x55fce6bc59a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fce6b3fb00 0x55fce6b64bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fce6b4b2d0 0x55fce6b5ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 --2- 172.21.15.63:0/3573614374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fce6b40420 0x55fce6b65110 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 >> 172.21.15.63:0/3573614374 conn(0x55fce6a66550 msgr2=0x55fce6b49e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa271600700 0 client.25241 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 shutdown_connections 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 1 -- 172.21.15.63:0/3573614374 wait complete. 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.890+0000 7fa29916eb80 20 client.25241 trim_cache size 0 max 16384 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 2, 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24508, 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:47:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 260.24754302500003, 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 24, 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 97, 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 96, 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 320.03808368300002 2024-04-25T18:47:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 msgr2=0x7fa26c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 0x7fa26c0a6390 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa288111140 tx=0x7fa274009310 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 msgr2=0x7fa28819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa27800e9c0 tx=0x7fa27800ecd0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 shutdown_connections 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa26c0a3ed0 0x7fa26c0a6390 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa28810f440 0x7fa28819fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa288074c70 0x7fa28819b3b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 --2- 172.21.15.63:0/87298448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa288075590 0x7fa28819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 >> 172.21.15.63:0/87298448 conn(0x7fa288079ba0 msgr2=0x7fa2880ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 shutdown_connections 2024-04-25T18:47:10.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:10.891+0000 7fa28e800700 1 -- 172.21.15.63:0/87298448 wait complete. 2024-04-25T18:47:11.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.i status 2024-04-25T18:47:12.240 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:12 smithi070 ceph-mon[32820]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:12.798 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:12 smithi063 ceph-mon[28006]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:12 smithi114 ceph-mon[32988]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:14.582 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:14 smithi063 ceph-mon[28006]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-25T18:47:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:14 smithi070 ceph-mon[32820]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 -- 172.21.15.63:0/1691374531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1678103990 msgr2=0x7f167806b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 --2- 172.21.15.63:0/1691374531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1678103990 0x7f167806b110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1668009990 tx=0x7f1668009ca0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 -- 172.21.15.63:0/1691374531 shutdown_connections 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 --2- 172.21.15.63:0/1691374531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 0x7f1678111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 --2- 172.21.15.63:0/1691374531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1678103990 0x7f167806b110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 --2- 172.21.15.63:0/1691374531 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f167806b650 0x7f167806bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 -- 172.21.15.63:0/1691374531 >> 172.21.15.63:0/1691374531 conn(0x7f16780788f0 msgr2=0x7f1678078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 -- 172.21.15.63:0/1691374531 shutdown_connections 2024-04-25T18:47:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.746+0000 7f167cc00700 1 -- 172.21.15.63:0/1691374531 wait complete. 2024-04-25T18:47:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.747+0000 7f167cc00700 1 Processor -- start 2024-04-25T18:47:14.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.747+0000 7f167cc00700 1 -- start start 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 0x7f167819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 0x7f167819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 0x7f167819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1678114e40 con 0x7f167810f390 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1678114cc0 con 0x7f167806b650 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f167cc00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1678114fc0 con 0x7f1678103990 2024-04-25T18:47:14.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f1675400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 0x7f167819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f1675e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 0x7f167819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f1676800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 0x7f167819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f1675400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 0x7f167819b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47472/0 (socket says 172.21.15.63:47472) 2024-04-25T18:47:14.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.748+0000 7f1675400700 1 -- 172.21.15.63:0/4252408958 learned_addr learned my addr 172.21.15.63:0/4252408958 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 -- 172.21.15.63:0/4252408958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 msgr2=0x7f167819b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 0x7f167819b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 -- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 msgr2=0x7f167819fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 0x7f167819fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 -- 172.21.15.63:0/4252408958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16781a02e0 con 0x7f167806b650 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f1675e00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 0x7f167819b2d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f16680098e0 tx=0x7f16680057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f166ea00700 1 -- 172.21.15.63:0/4252408958 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668027070 con 0x7f167806b650 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f166ea00700 1 -- 172.21.15.63:0/4252408958 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1668022470 con 0x7f167806b650 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.749+0000 7f166ea00700 1 -- 172.21.15.63:0/4252408958 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668017650 con 0x7f167806b650 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.750+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16781a0570 con 0x7f167806b650 2024-04-25T18:47:14.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.750+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16781a7e50 con 0x7f167806b650 2024-04-25T18:47:14.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.753+0000 7f166ea00700 1 -- 172.21.15.63:0/4252408958 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f166802a070 con 0x7f167806b650 2024-04-25T18:47:14.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.755+0000 7f166ea00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 0x7f165c0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.755+0000 7f166ea00700 1 -- 172.21.15.63:0/4252408958 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f16680d1890 con 0x7f167806b650 2024-04-25T18:47:14.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.755+0000 7f1675400700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 0x7f165c0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.756+0000 7f1675400700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 0x7f165c0a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1678079110 tx=0x7f1664007480 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.756+0000 7f168755eb80 1 Processor -- start 2024-04-25T18:47:14.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 -- start start 2024-04-25T18:47:14.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e26544c10 0x563e2654f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e2654fab0 0x563e26551ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e26497e40 con 0x563e2654fab0 2024-04-25T18:47:14.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563e2643e6e0 con 0x563e265442f0 2024-04-25T18:47:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f168755eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e26493e00 con 0x563e26544c10 2024-04-25T18:47:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f1675e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f1675e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44588/0 (socket says 172.21.15.63:44588) 2024-04-25T18:47:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.757+0000 7f1675400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e26544c10 0x563e2654f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 -- 172.21.15.63:0/1143704408 learned_addr learned my addr 172.21.15.63:0/1143704408 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1676800700 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e2654fab0 0x563e26551ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 -- 172.21.15.63:0/1143704408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e26544c10 msgr2=0x563e2654f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e26544c10 0x563e2654f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 -- 172.21.15.63:0/1143704408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e2654fab0 msgr2=0x563e26551ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e2654fab0 0x563e26551ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 -- 172.21.15.63:0/1143704408 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e26434630 con 0x563e265442f0 2024-04-25T18:47:14.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1675e00700 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1668099fd0 tx=0x7f1668022cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0862c194f5ce032 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1663400700 1 -- 172.21.15.63:0/1143704408 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166809d600 con 0x563e265442f0 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.758+0000 7f1663400700 1 -- 172.21.15.63:0/1143704408 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680a19b0 con 0x563e265442f0 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f1663400700 1 -- 172.21.15.63:0/1143704408 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166809d860 con 0x563e265442f0 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 -- 172.21.15.63:0/1143704408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 msgr2=0x563e265446d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1668099fd0 tx=0x7f1668022cc0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 -- 172.21.15.63:0/1143704408 shutdown_connections 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e2654fab0 0x563e26551ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e265442f0 0x563e265446d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 --2- 172.21.15.63:0/1143704408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e26544c10 0x563e2654f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.759+0000 7f168755eb80 1 -- 172.21.15.63:0/1143704408 >> 172.21.15.63:0/1143704408 conn(0x563e26473550 msgr2=0x563e26540790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.760+0000 7f168755eb80 1 -- 172.21.15.63:0/1143704408 shutdown_connections 2024-04-25T18:47:14.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.760+0000 7f168755eb80 1 -- 172.21.15.63:0/1143704408 wait complete. 2024-04-25T18:47:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.760+0000 7f168755eb80 1 -- start start 2024-04-25T18:47:14.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 Processor -- start 2024-04-25T18:47:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e26544c10 0x563e2654c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e2654fab0 0x563e2654cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e2643e6e0 con 0x563e265442f0 2024-04-25T18:47:14.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563e26451810 con 0x563e26544c10 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e265523e0 con 0x563e2654fab0 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.761+0000 7f168755eb80 10 client.0 _ll_register_callbacks cb 0x563e264306d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40358/0 (socket says 172.21.15.63:40358) 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 -- 172.21.15.63:0/266814598 learned_addr learned my addr 172.21.15.63:0/266814598 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675400700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e26544c10 0x563e2654c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1676800700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e2654fab0 0x563e2654cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 -- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e2654fab0 msgr2=0x563e2654cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e2654fab0 0x563e2654cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 -- 172.21.15.63:0/266814598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e26544c10 msgr2=0x563e2654c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e26544c10 0x563e2654c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.762+0000 7f1675e00700 1 -- 172.21.15.63:0/266814598 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e263e5880 con 0x563e265442f0 2024-04-25T18:47:14.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f1675e00700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16680a4080 tx=0x7f16680096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6bb96ea107c6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f1657e00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166809d600 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16680a19b0 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f166809da70 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f16680bc320 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f168755eb80 10 client.15105 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.763+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e2642eaa0 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680a19b0 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f16680cf1e0 con 0x563e265442f0 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 10 client.15105 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 10 client.15105 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 20 client.15105 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:14.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e2655ee10 0x563e26560280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 4 client.15105 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.764+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e264a5630 con 0x563e2655ee10 2024-04-25T18:47:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.765+0000 7f1675400700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e2655ee10 0x563e26560280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.766+0000 7f1675400700 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e2655ee10 0x563e26560280 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c6553960f1298f9 server_cookie=3f1b0ed01667bdcd in_seq=0 out_seq=0 2024-04-25T18:47:14.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.766+0000 7f1657e00700 10 client.15105 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:47:14.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.766+0000 7f1657e00700 1 -- 172.21.15.63:0/266814598 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e264a5630 con 0x563e2655ee10 2024-04-25T18:47:14.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.766+0000 7f1657e00700 10 client.15105 handle_command_reply: tid=0 2024-04-25T18:47:14.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 client.15105 shutdown 2024-04-25T18:47:14.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 msgr2=0x563e2654f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16680a4080 tx=0x7f16680096e0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 shutdown_connections 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e2655ee10 0x563e26560280 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e2654fab0 0x563e2654cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e265442f0 0x563e2654f2a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 --2- 172.21.15.63:0/266814598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e26544c10 0x563e2654c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f1657e00700 0 client.15105 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 >> 172.21.15.63:0/266814598 conn(0x563e26473550 msgr2=0x563e26546c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 shutdown_connections 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 1 -- 172.21.15.63:0/266814598 wait complete. 2024-04-25T18:47:14.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.767+0000 7f168755eb80 20 client.15105 trim_cache size 0 max 16384 2024-04-25T18:47:14.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 Processor -- start 2024-04-25T18:47:14.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 -- start start 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e2654ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e26549d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e2655e5c0 con 0x563e26544c10 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563e2655ec90 con 0x563e2654fab0 2024-04-25T18:47:14.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.800+0000 7f168755eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e2655f360 con 0x563e265442f0 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e2654ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40360/0 (socket says 172.21.15.63:40360) 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675400700 1 -- 172.21.15.63:0/3242918625 learned_addr learned my addr 172.21.15.63:0/3242918625 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1676800700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e26549d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 -- 172.21.15.63:0/3242918625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 msgr2=0x563e26549d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e26549d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 -- 172.21.15.63:0/3242918625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 msgr2=0x563e26549840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675e00700 1 -- 172.21.15.63:0/3242918625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e26540950 con 0x563e265442f0 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1675400700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:14.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.801+0000 7f1676800700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e26549d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.802+0000 7f1675e00700 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e2654ca90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1668009750 tx=0x7f1668022cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=891ff5fdfe206d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.802+0000 7f1662a00700 1 -- 172.21.15.63:0/3242918625 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166809d600 con 0x563e265442f0 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.802+0000 7f1662a00700 1 -- 172.21.15.63:0/3242918625 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680a19b0 con 0x563e265442f0 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.802+0000 7f1662a00700 1 -- 172.21.15.63:0/3242918625 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166809b900 con 0x563e265442f0 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 -- 172.21.15.63:0/3242918625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 msgr2=0x563e2654ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e2654ca90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1668009750 tx=0x7f1668022cc0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 -- 172.21.15.63:0/3242918625 shutdown_connections 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e2654ca90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e26549d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 --2- 172.21.15.63:0/3242918625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26549840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 -- 172.21.15.63:0/3242918625 >> 172.21.15.63:0/3242918625 conn(0x563e26473550 msgr2=0x563e2654c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 -- 172.21.15.63:0/3242918625 shutdown_connections 2024-04-25T18:47:14.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.803+0000 7f168755eb80 1 -- 172.21.15.63:0/3242918625 wait complete. 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.804+0000 7f168755eb80 1 -- start start 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.804+0000 7f168755eb80 1 Processor -- start 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e265654f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26565a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e26560100 con 0x563e26544c10 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563e2655fa30 con 0x563e2654fab0 2024-04-25T18:47:14.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563e265607d0 con 0x563e265442f0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f168755eb80 10 client.0 _ll_register_callbacks cb 0x563e264306d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1675e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e265654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44612/0 (socket says 172.21.15.63:44612) 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 -- 172.21.15.63:0/2542063567 learned_addr learned my addr 172.21.15.63:0/2542063567 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1675400700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26565a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 -- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 msgr2=0x563e265654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e265654f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 -- 172.21.15.63:0/2542063567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 msgr2=0x563e26565a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26565a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.805+0000 7f1676800700 1 -- 172.21.15.63:0/2542063567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e265c5480 con 0x563e2654fab0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f1676800700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1670007bf0 tx=0x7f1670009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af180be26b0b17c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f1657e00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:14.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670004db0 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16680a19b0 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f1670003a00 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1670003c00 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f168755eb80 10 client.25259 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.806+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563e265c6750 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16680a19b0 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f16700043a0 con 0x563e2654fab0 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 10 client.25259 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 10 client.25259 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 20 client.25259 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e265cb350 0x563e265ca060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 4 client.25259 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:47:14.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "status"}) -- 0x563e2649f6d0 con 0x563e265cb350 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.807+0000 7f1675e00700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e265cb350 0x563e265ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f1675e00700 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e265cb350 0x563e265ca060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2484cddb83bb7cb5 server_cookie=4355aa974d0fe050 in_seq=0 out_seq=0 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f1657e00700 10 client.25259 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f1657e00700 1 -- 172.21.15.63:0/2542063567 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563e2649f6d0 con 0x563e265cb350 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f1657e00700 10 client.25259 handle_command_reply: tid=0 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 client.25259 shutdown 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 msgr2=0x563e265c4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1670007bf0 tx=0x7f1670009510 comp rx=0 tx=0).stop 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 shutdown_connections 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x563e265cb350 0x563e265ca060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563e265442f0 0x563e265654f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563e2654fab0 0x563e265c4f40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 --2- 172.21.15.63:0/2542063567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e26544c10 0x563e26565a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 >> 172.21.15.63:0/2542063567 conn(0x563e26473550 msgr2=0x563e2654ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.808+0000 7f1657e00700 0 client.25259 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 shutdown_connections 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f168755eb80 1 -- 172.21.15.63:0/2542063567 wait complete. 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f168755eb80 20 client.25259 trim_cache size 0 max 16384 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 1, 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24520, 2024-04-25T18:47:14.813 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 265.17761819100002, 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 24, 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 97, 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 96, 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 318.66281767300001 2024-04-25T18:47:14.814 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 msgr2=0x7f165c0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 0x7f165c0a6410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1678079110 tx=0x7f1664007480 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 msgr2=0x7f167819b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.809+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 0x7f167819b2d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f16680098e0 tx=0x7f16680057c0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 shutdown_connections 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f165c0a3f50 0x7f165c0a6410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f167810f390 0x7f167819fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f167806b650 0x7f167819b2d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 --2- 172.21.15.63:0/4252408958 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1678103990 0x7f167819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 >> 172.21.15.63:0/4252408958 conn(0x7f16780788f0 msgr2=0x7f1678100d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:14.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 shutdown_connections 2024-04-25T18:47:14.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:14.810+0000 7f167cc00700 1 -- 172.21.15.63:0/4252408958 wait complete. 2024-04-25T18:47:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:14 smithi114 ceph-mon[32988]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-04-25T18:47:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 -- 172.21.15.63:0/4219961852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076480 msgr2=0x7f9d48076860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 --2- 172.21.15.63:0/4219961852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076480 0x7f9d48076860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009900 tx=0x7f9d3c02a3f0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.496 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 -- 172.21.15.63:0/4219961852 shutdown_connections 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 --2- 172.21.15.63:0/4219961852 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48112c40 0x7f9d48115030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 --2- 172.21.15.63:0/4219961852 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48076da0 0x7f9d48112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 --2- 172.21.15.63:0/4219961852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076480 0x7f9d48076860 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 -- 172.21.15.63:0/4219961852 >> 172.21.15.63:0/4219961852 conn(0x7f9d480ffa70 msgr2=0x7f9d48101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 -- 172.21.15.63:0/4219961852 shutdown_connections 2024-04-25T18:47:15.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.495+0000 7f9d50600640 1 -- 172.21.15.63:0/4219961852 wait complete. 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.496+0000 7f9d50600640 1 Processor -- start 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.496+0000 7f9d50600640 1 -- start start 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 0x7f9d481a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48112c40 0x7f9d4819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d48118690 con 0x7f9d48076da0 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d48118510 con 0x7f9d48076480 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d50600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d48118810 con 0x7f9d48112c40 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d4d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d4dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 0x7f9d481a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d4d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40388/0 (socket says 172.21.15.63:40388) 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d4d200640 1 -- 172.21.15.63:0/1115726381 learned_addr learned my addr 172.21.15.63:0/1115726381 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.497+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48112c40 0x7f9d4819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 -- 172.21.15.63:0/1115726381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48112c40 msgr2=0x7f9d4819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48112c40 0x7f9d4819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 -- 172.21.15.63:0/1115726381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 msgr2=0x7f9d481a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 0x7f9d481a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 -- 172.21.15.63:0/1115726381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4819f1f0 con 0x7f9d48076da0 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 0x7f9d481a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400e9f0 tx=0x7f9d3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d46000640 1 -- 172.21.15.63:0/1115726381 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400cd80 con 0x7f9d48076da0 2024-04-25T18:47:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d46000640 1 -- 172.21.15.63:0/1115726381 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d340108e0 con 0x7f9d48076da0 2024-04-25T18:47:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4819f4e0 con 0x7f9d48076da0 2024-04-25T18:47:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d46000640 1 -- 172.21.15.63:0/1115726381 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34018980 con 0x7f9d48076da0 2024-04-25T18:47:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.498+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d481ab780 con 0x7f9d48076da0 2024-04-25T18:47:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 Processor -- start 2024-04-25T18:47:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 -- start start 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd521650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd523f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562fd47c900 con 0x5562fd5169b0 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562fd422ef0 con 0x5562fd521b90 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d53421740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562fd4788c0 con 0x5562fd5172d0 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.502+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40402/0 (socket says 172.21.15.63:40402) 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 -- 172.21.15.63:0/1396818698 learned_addr learned my addr 172.21.15.63:0/1396818698 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd521650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 -- 172.21.15.63:0/1396818698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 msgr2=0x5562fd521650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd521650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 -- 172.21.15.63:0/1396818698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 msgr2=0x5562fd523f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd523f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 -- 172.21.15.63:0/1396818698 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fd18f210 con 0x5562fd5169b0 2024-04-25T18:47:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.503+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009f30 tx=0x7f9d400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ff9247d7bc9216 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d3aa00640 1 -- 172.21.15.63:0/1396818698 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40004d20 con 0x5562fd5169b0 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d3aa00640 1 -- 172.21.15.63:0/1396818698 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40004050 con 0x5562fd5169b0 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d3aa00640 1 -- 172.21.15.63:0/1396818698 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40005280 con 0x5562fd5169b0 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d46000640 1 -- 172.21.15.63:0/1115726381 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f9d34010430 con 0x7f9d48076da0 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d53421740 1 -- 172.21.15.63:0/1396818698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 msgr2=0x5562fd516d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.504+0000 7f9d53421740 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009f30 tx=0x7f9d400118d0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 -- 172.21.15.63:0/1396818698 shutdown_connections 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd523f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd521650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 --2- 172.21.15.63:0/1396818698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd516d90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 -- 172.21.15.63:0/1396818698 >> 172.21.15.63:0/1396818698 conn(0x5562fd512ec0 msgr2=0x5562fd5132a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 -- 172.21.15.63:0/1396818698 shutdown_connections 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 -- 172.21.15.63:0/1396818698 wait complete. 2024-04-25T18:47:15.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 -- start start 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.505+0000 7f9d53421740 1 Processor -- start 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd51eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd51f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd51f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562fd5247c0 con 0x5562fd5169b0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562fd436310 con 0x5562fd521b90 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562fd524940 con 0x5562fd5172d0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d53421740 10 client.0 _ll_register_callbacks cb 0x5562fd404cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd51eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d46000640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 0x7f9d300a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd51eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40418/0 (socket says 172.21.15.63:40418) 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.506+0000 7f9d4e600640 1 -- 172.21.15.63:0/2089821608 learned_addr learned my addr 172.21.15.63:0/2089821608 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd51f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d46000640 1 -- 172.21.15.63:0/1115726381 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d340d0de0 con 0x7f9d48076da0 2024-04-25T18:47:15.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4d200640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd51f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 0x7f9d300a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 -- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 msgr2=0x5562fd51f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd51f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 -- 172.21.15.63:0/2089821608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 msgr2=0x5562fd51eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd51eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 -- 172.21.15.63:0/2089821608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fd25b620 con 0x5562fd5172d0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4d200640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd51f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.507+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd51f090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c02a910 tx=0x7f9d3c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a146a3b9c43d629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d2f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 0x7f9d300a6200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009f00 tx=0x7f9d40009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c035040 con 0x5562fd5172d0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d40004050 con 0x5562fd5172d0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d3c004760 con 0x5562fd5172d0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d3c004960 con 0x5562fd5172d0 2024-04-25T18:47:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d53421740 10 client.25288 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562fd530a50 con 0x5562fd5172d0 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.508+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40004050 con 0x5562fd5172d0 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9d3c03c440 con 0x5562fd5172d0 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 10 client.25288 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 10 client.25288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 20 client.25288 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd531f10 0x5562fd534300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 4 client.25288 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562fd4908b0 con 0x5562fd531f10 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.509+0000 7f9d4d200640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd531f10 0x5562fd534300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.510+0000 7f9d4d200640 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd531f10 0x5562fd534300 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df5026d2910ca78c server_cookie=3474d1ce71220a4c in_seq=0 out_seq=0 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.510+0000 7f9d2f400640 10 client.25288 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d2f400640 1 -- 172.21.15.63:0/2089821608 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562fd4908b0 con 0x5562fd531f10 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d2f400640 10 client.25288 handle_command_reply: tid=0 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 client.25288 shutdown 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 msgr2=0x5562fd51f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd51f090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c02a910 tx=0x7f9d3c005270 comp rx=0 tx=0).stop 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 shutdown_connections 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd531f10 0x5562fd534300 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd51f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd51f090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d2f400640 0 client.25288 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 --2- 172.21.15.63:0/2089821608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd51eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 >> 172.21.15.63:0/2089821608 conn(0x5562fd512ec0 msgr2=0x5562fd518bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 shutdown_connections 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 1 -- 172.21.15.63:0/2089821608 wait complete. 2024-04-25T18:47:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.511+0000 7f9d53421740 20 client.25288 trim_cache size 0 max 16384 2024-04-25T18:47:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.530+0000 7f9d53421740 1 Processor -- start 2024-04-25T18:47:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.530+0000 7f9d53421740 1 -- start start 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.530+0000 7f9d53421740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.530+0000 7f9d53421740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5127a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d53421740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d53421740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b6970 con 0x5562fd5169b0 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d53421740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b4240 con 0x5562fd521b90 2024-04-25T18:47:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d53421740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b2090 con 0x5562fd5172d0 2024-04-25T18:47:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4dc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5127a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44644/0 (socket says 172.21.15.63:44644) 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4dc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40428/0 (socket says 172.21.15.63:40428) 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4e600640 1 -- 172.21.15.63:0/233866544 learned_addr learned my addr 172.21.15.63:0/233866544 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 -- 172.21.15.63:0/233866544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 msgr2=0x5562fd539ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 -- 172.21.15.63:0/233866544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 msgr2=0x5562fd52d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4d200640 1 -- 172.21.15.63:0/233866544 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562fd512ce0 con 0x5562fd5172d0 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.531+0000 7f9d4e600640 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.532+0000 7f9d4d200640 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5127a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ce340 tx=0x7f9d340d31f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d1b942ca6ff126 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.532+0000 7f9d3a000640 1 -- 172.21.15.63:0/233866544 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d340bd0b0 con 0x5562fd5172d0 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.532+0000 7f9d3a000640 1 -- 172.21.15.63:0/233866544 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3409ac10 con 0x5562fd5172d0 2024-04-25T18:47:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.532+0000 7f9d3a000640 1 -- 172.21.15.63:0/233866544 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34018b20 con 0x5562fd5172d0 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 -- 172.21.15.63:0/233866544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 msgr2=0x5562fd5127a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5127a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ce340 tx=0x7f9d340d31f0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 -- 172.21.15.63:0/233866544 shutdown_connections 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd539ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5127a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 --2- 172.21.15.63:0/233866544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 -- 172.21.15.63:0/233866544 >> 172.21.15.63:0/233866544 conn(0x5562fd512ec0 msgr2=0x5562fd520c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 -- 172.21.15.63:0/233866544 shutdown_connections 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.533+0000 7f9d53421740 1 -- 172.21.15.63:0/233866544 wait complete. 2024-04-25T18:47:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 -- start start 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 Processor -- start 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd527940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b2700 con 0x5562fd5169b0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b3b70 con 0x5562fd521b90 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481b2dd0 con 0x5562fd5172d0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.534+0000 7f9d53421740 10 client.0 _ll_register_callbacks cb 0x7f9d3c004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47566/0 (socket says 172.21.15.63:47566) 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 -- 172.21.15.63:0/1232985452 learned_addr learned my addr 172.21.15.63:0/1232985452 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4e600640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd527940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 -- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 msgr2=0x5562fd527940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd527940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 -- 172.21.15.63:0/1232985452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 msgr2=0x5562fd52a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 -- 172.21.15.63:0/1232985452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562fd528170 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d4d200640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ce310 tx=0x7f9d34018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5290c519f43f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.535+0000 7f9d2f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d34010a90 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d3409c180 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d3409ac10 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9d340b6020 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 10 client.25294 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562fd598500 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3409ac10 con 0x5562fd5172d0 2024-04-25T18:47:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f9d34098950 con 0x5562fd5172d0 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 10 client.25294 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 10 client.25294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 20 client.25294 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd598c50 0x5562fd59b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 4 client.25294 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.536+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562fd48a320 con 0x5562fd598c50 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.537+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd598c50 0x5562fd59b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.537+0000 7f9d4dc00640 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd598c50 0x5562fd59b070 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61093e40c788abec server_cookie=7f7cf419cb0b742 in_seq=0 out_seq=0 2024-04-25T18:47:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.537+0000 7f9d2f400640 10 client.25294 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d2f400640 1 -- 172.21.15.63:0/1232985452 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562fd48a320 con 0x5562fd598c50 2024-04-25T18:47:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d2f400640 10 client.25294 handle_command_reply: tid=0 2024-04-25T18:47:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 client.25294 shutdown 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 msgr2=0x5562fd5273a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ce310 tx=0x7f9d34018420 comp rx=0 tx=0).stop 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 shutdown_connections 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562fd598c50 0x5562fd59b070 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562fd521b90 0x5562fd527940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562fd5172d0 0x5562fd5273a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d2f400640 0 client.25294 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 --2- 172.21.15.63:0/1232985452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562fd5169b0 0x5562fd52a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 >> 172.21.15.63:0/1232985452 conn(0x5562fd512ec0 msgr2=0x5562fd520c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 shutdown_connections 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 1 -- 172.21.15.63:0/1232985452 wait complete. 2024-04-25T18:47:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.546+0000 7f9d53421740 20 client.25294 trim_cache size 0 max 16384 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 msgr2=0x7f9d300a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 0x7f9d300a6200 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009f00 tx=0x7f9d40009500 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 msgr2=0x7f9d481a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400e9f0 tx=0x7f9d3400eeb0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 shutdown_connections 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d300a3d40 0x7f9d300a6200 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d48112c40 0x7f9d4819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d48076da0 0x7f9d481a4e00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 --2- 172.21.15.63:0/1115726381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d48076480 0x7f9d481a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.547+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 >> 172.21.15.63:0/1115726381 conn(0x7f9d480ffa70 msgr2=0x7f9d48101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.548+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 shutdown_connections 2024-04-25T18:47:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:15.548+0000 7f9d50600640 1 -- 172.21.15.63:0/1115726381 wait complete. 2024-04-25T18:47:15.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.b status 2024-04-25T18:47:16.299 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:16.678 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:16 smithi063 ceph-mon[28006]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:16 smithi070 ceph-mon[32820]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:16 smithi114 ceph-mon[32988]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T18:47:18.497 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:18 smithi063 ceph-mon[28006]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:18 smithi070 ceph-mon[32820]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 -- 172.21.15.63:0/3636835486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564103450 msgr2=0x7fe56410f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 --2- 172.21.15.63:0/3636835486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564103450 0x7fe56410f8e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00b0a0 tx=0x7fe55c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 -- 172.21.15.63:0/3636835486 shutdown_connections 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 --2- 172.21.15.63:0/3636835486 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564103450 0x7fe56410f8e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 --2- 172.21.15.63:0/3636835486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564108a80 0x7fe564108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 --2- 172.21.15.63:0/3636835486 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564102a90 0x7fe564102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 -- 172.21.15.63:0/3636835486 >> 172.21.15.63:0/3636835486 conn(0x7fe564079bf0 msgr2=0x7fe56407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 -- 172.21.15.63:0/3636835486 shutdown_connections 2024-04-25T18:47:18.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.791+0000 7fe56c600700 1 -- 172.21.15.63:0/3636835486 wait complete. 2024-04-25T18:47:18.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.793+0000 7fe56c600700 1 Processor -- start 2024-04-25T18:47:18.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.793+0000 7fe56c600700 1 -- start start 2024-04-25T18:47:18.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564102a90 0x7fe564074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564108a80 0x7fe5640757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe564112c80 con 0x7fe564103450 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe564112b00 con 0x7fe564102a90 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56c600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe564112e00 con 0x7fe564108a80 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe569200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe569c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564102a90 0x7fe564074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe569200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45894/0 (socket says 172.21.15.63:45894) 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe569200700 1 -- 172.21.15.63:0/2414883335 learned_addr learned my addr 172.21.15.63:0/2414883335 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:18.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.794+0000 7fe56a600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564108a80 0x7fe5640757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 -- 172.21.15.63:0/2414883335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564108a80 msgr2=0x7fe5640757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564108a80 0x7fe5640757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 -- 172.21.15.63:0/2414883335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564102a90 msgr2=0x7fe564074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564102a90 0x7fe564074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 -- 172.21.15.63:0/2414883335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5641a7610 con 0x7fe564103450 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe569200700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe55000e9c0 tx=0x7fe55000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.795+0000 7fe562000700 1 -- 172.21.15.63:0/2414883335 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55000cbd0 con 0x7fe564103450 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.796+0000 7fe562000700 1 -- 172.21.15.63:0/2414883335 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe55000ee60 con 0x7fe564103450 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.796+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5641a7900 con 0x7fe564103450 2024-04-25T18:47:18.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.796+0000 7fe562000700 1 -- 172.21.15.63:0/2414883335 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5500055c0 con 0x7fe564103450 2024-04-25T18:47:18.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.796+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641a7df0 con 0x7fe564103450 2024-04-25T18:47:18.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.799+0000 7fe562000700 1 -- 172.21.15.63:0/2414883335 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fe550004750 con 0x7fe564103450 2024-04-25T18:47:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.801+0000 7fe576ecdb80 1 Processor -- start 2024-04-25T18:47:18.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe562000700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 0x7fe54c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 -- start start 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe562000700 1 -- 172.21.15.63:0/2414883335 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fe5500d0de0 con 0x7fe564103450 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 0x55fde7653b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe56a600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 0x7fe54c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde76540a0 0x55fde764d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde764e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde759fe40 con 0x55fde76540a0 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fde75466e0 con 0x55fde764def0 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.802+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fde759be00 con 0x55fde7653780 2024-04-25T18:47:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe56a600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde764e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 0x55fde7653b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe56a600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde764e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44714/0 (socket says 172.21.15.63:44714) 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe56a600700 1 -- 172.21.15.63:0/3440782427 learned_addr learned my addr 172.21.15.63:0/3440782427 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569200700 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde76540a0 0x55fde764d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 -- 172.21.15.63:0/3440782427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 msgr2=0x55fde764e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde764e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 -- 172.21.15.63:0/3440782427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde76540a0 msgr2=0x55fde764d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde76540a0 0x55fde764d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe56a600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 0x7fe54c0a6390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d010 tx=0x7fe55c00bb90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.803+0000 7fe569c00700 1 -- 172.21.15.63:0/3440782427 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde753c630 con 0x55fde7653780 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.804+0000 7fe569c00700 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 0x55fde7653b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe558009cb0 tx=0x7fe5580058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f6c0bf7df5fda0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.804+0000 7fe556a00700 1 -- 172.21.15.63:0/3440782427 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558011c20 con 0x55fde7653780 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.804+0000 7fe556a00700 1 -- 172.21.15.63:0/3440782427 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe558004500 con 0x55fde7653780 2024-04-25T18:47:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.804+0000 7fe556a00700 1 -- 172.21.15.63:0/3440782427 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558005280 con 0x55fde7653780 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 -- 172.21.15.63:0/3440782427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 msgr2=0x55fde7653b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 0x55fde7653b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe558009cb0 tx=0x7fe5580058e0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 -- 172.21.15.63:0/3440782427 shutdown_connections 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7653780 0x55fde7653b60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde764e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/3440782427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde76540a0 0x55fde764d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.805+0000 7fe576ecdb80 1 -- 172.21.15.63:0/3440782427 >> 172.21.15.63:0/3440782427 conn(0x55fde757bf00 msgr2=0x55fde76488a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.806+0000 7fe576ecdb80 1 -- 172.21.15.63:0/3440782427 shutdown_connections 2024-04-25T18:47:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.806+0000 7fe576ecdb80 1 -- 172.21.15.63:0/3440782427 wait complete. 2024-04-25T18:47:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.806+0000 7fe576ecdb80 1 -- start start 2024-04-25T18:47:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.806+0000 7fe576ecdb80 1 Processor -- start 2024-04-25T18:47:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde7659bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 0x55fde76578d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde765a920 con 0x55fde7653780 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fde7559810 con 0x55fde764def0 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fde765aaa0 con 0x55fde7657450 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe576ecdb80 10 client.0 _ll_register_callbacks cb 0x55fde75386d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe569200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe56a600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 0x55fde76578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe569200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45922/0 (socket says 172.21.15.63:45922) 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.807+0000 7fe569200700 1 -- 172.21.15.63:0/2339742345 learned_addr learned my addr 172.21.15.63:0/2339742345 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:18.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe569c00700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde7659bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 -- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 msgr2=0x55fde7659bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde7659bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 -- 172.21.15.63:0/2339742345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 msgr2=0x55fde7656f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 -- 172.21.15.63:0/2339742345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde74ed880 con 0x55fde7657450 2024-04-25T18:47:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe569200700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe56a600700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 0x55fde76578d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009d10 tx=0x7fe55c00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e77b38e7f7c0eaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.808+0000 7fe54b400700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c009440 con 0x55fde7657450 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fe55c0078c0 con 0x55fde7657450 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe558004500 con 0x55fde7657450 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe55c038050 con 0x55fde7657450 2024-04-25T18:47:18.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe576ecdb80 10 client.25306 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fde7536aa0 con 0x55fde7657450 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.809+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe558004500 con 0x55fde7657450 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe55c011070 con 0x55fde7657450 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 10 client.25306 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 10 client.25306 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 20 client.25306 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde7667960 0x55fde7668dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 4 client.25306 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fde75ad630 con 0x55fde7667960 2024-04-25T18:47:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.810+0000 7fe569c00700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde7667960 0x55fde7668dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.811+0000 7fe569c00700 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde7667960 0x55fde7668dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4eb861ea1b6862fc server_cookie=2e82516f21580a17 in_seq=0 out_seq=0 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.811+0000 7fe54b400700 10 client.25306 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe54b400700 1 -- 172.21.15.63:0/2339742345 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fde75ad630 con 0x55fde7667960 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe54b400700 10 client.25306 handle_command_reply: tid=0 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe576ecdb80 1 client.25306 shutdown 2024-04-25T18:47:18.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 msgr2=0x55fde76578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 0x55fde76578d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009d10 tx=0x7fe55c00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.812+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 shutdown_connections 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde7667960 0x55fde7668dd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde7657450 0x55fde76578d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde764def0 0x55fde7659bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/2339742345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde7656f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 >> 172.21.15.63:0/2339742345 conn(0x55fde757bf00 msgr2=0x55fde7647e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe54b400700 0 client.25306 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:18.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 shutdown_connections 2024-04-25T18:47:18.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 1 -- 172.21.15.63:0/2339742345 wait complete. 2024-04-25T18:47:18.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.813+0000 7fe576ecdb80 20 client.25306 trim_cache size 0 max 16384 2024-04-25T18:47:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:18 smithi114 ceph-mon[32988]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:18.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 Processor -- start 2024-04-25T18:47:18.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 -- start start 2024-04-25T18:47:18.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 0x55fde7663d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 0x55fde7656cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde7667110 con 0x55fde7657450 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fde76677e0 con 0x55fde7653780 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.836+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fde7667eb0 con 0x55fde764def0 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41312/0 (socket says 172.21.15.63:41312) 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 -- 172.21.15.63:0/1913481746 learned_addr learned my addr 172.21.15.63:0/1913481746 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569200700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 0x55fde7663d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe56a600700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 0x55fde7656cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 -- 172.21.15.63:0/1913481746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 msgr2=0x55fde7663d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 0x55fde7663d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 -- 172.21.15.63:0/1913481746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 msgr2=0x55fde7656cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 0x55fde7656cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569c00700 1 -- 172.21.15.63:0/1913481746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde7659800 con 0x55fde764def0 2024-04-25T18:47:18.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe569200700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 0x55fde7663d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:18.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.837+0000 7fe56a600700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 0x55fde7656cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:18.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.838+0000 7fe569c00700 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe558000c00 tx=0x7fe558009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b03404f80d97199e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.838+0000 7fe556000700 1 -- 172.21.15.63:0/1913481746 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55802c070 con 0x55fde764def0 2024-04-25T18:47:18.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.838+0000 7fe556000700 1 -- 172.21.15.63:0/1913481746 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe558004ad0 con 0x55fde764def0 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.838+0000 7fe556000700 1 -- 172.21.15.63:0/1913481746 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558027470 con 0x55fde764def0 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- 172.21.15.63:0/1913481746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 msgr2=0x55fde7652fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe558000c00 tx=0x7fe558009710 comp rx=0 tx=0).stop 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- 172.21.15.63:0/1913481746 shutdown_connections 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7652fe0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7657450 0x55fde7656cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/1913481746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7653780 0x55fde7663d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- 172.21.15.63:0/1913481746 >> 172.21.15.63:0/1913481746 conn(0x55fde757bf00 msgr2=0x55fde7644ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- 172.21.15.63:0/1913481746 shutdown_connections 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- 172.21.15.63:0/1913481746 wait complete. 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 -- start start 2024-04-25T18:47:18.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.839+0000 7fe576ecdb80 1 Processor -- start 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde76476a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7657450 0x55fde7647be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde7668c50 con 0x55fde7653780 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fde7668580 con 0x55fde7657450 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fde7669320 con 0x55fde764def0 2024-04-25T18:47:18.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe576ecdb80 10 client.0 _ll_register_callbacks cb 0x55fde75386d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe569c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.840+0000 7fe569200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde76476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41326/0 (socket says 172.21.15.63:41326) 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 -- 172.21.15.63:0/4056566956 learned_addr learned my addr 172.21.15.63:0/4056566956 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe56a600700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7657450 0x55fde7647be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 -- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7657450 msgr2=0x55fde7647be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7657450 0x55fde7647be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 -- 172.21.15.63:0/4056566956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 msgr2=0x55fde76476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde76476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 -- 172.21.15.63:0/4056566956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde7648420 con 0x55fde764def0 2024-04-25T18:47:18.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe569c00700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe5580118d0 tx=0x7fe558009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95f4d8cfa3ef0ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:18.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.841+0000 7fe54b400700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:18.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe558037060 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe558004ad0 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fe558011a20 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe55802c040 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe576ecdb80 10 client.25312 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.842+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fde76cea90 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe558004ad0 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe558018070 con 0x55fde764def0 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 10 client.25312 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 10 client.25312 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 20 client.25312 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde76d2f30 0x55fde76d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 4 client.25312 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "status"}) -- 0x55fde75a76d0 con 0x55fde76d2f30 2024-04-25T18:47:18.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.843+0000 7fe569200700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde76d2f30 0x55fde76d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:18.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe569200700 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde76d2f30 0x55fde76d1c40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42e0ada04df669d server_cookie=d562c21bb7e3db77 in_seq=0 out_seq=0 2024-04-25T18:47:18.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe54b400700 10 client.25312 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:18.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe54b400700 1 -- 172.21.15.63:0/4056566956 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fde75a76d0 con 0x55fde76d2f30 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe54b400700 10 client.25312 handle_command_reply: tid=0 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 client.25312 shutdown 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 msgr2=0x55fde7672610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe5580118d0 tx=0x7fe558009490 comp rx=0 tx=0).stop 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 shutdown_connections 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fde76d2f30 0x55fde76d1c40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fde764def0 0x55fde7672610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fde7657450 0x55fde7647be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 --2- 172.21.15.63:0/4056566956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde7653780 0x55fde76476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 >> 172.21.15.63:0/4056566956 conn(0x55fde757bf00 msgr2=0x55fde7656cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.844+0000 7fe54b400700 0 client.25312 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.845+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 shutdown_connections 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.845+0000 7fe576ecdb80 1 -- 172.21.15.63:0/4056566956 wait complete. 2024-04-25T18:47:18.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.845+0000 7fe576ecdb80 20 client.25312 trim_cache size 0 max 16384 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "cluster_fsid": "e90c993c-0331-11ef-bc93-c7b262605968", 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "whoami": 0, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "id": 24533, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "want_state": "up:active", 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "up:active", 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "fs_name": "cephfs", 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "rank_uptime": 285.949835452, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "mdsmap_epoch": 24, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch": 97, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_epoch_barrier": 96, 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 309.86620156800001 2024-04-25T18:47:18.848 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 msgr2=0x7fe54c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 0x7fe54c0a6390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00d010 tx=0x7fe55c00bb90 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 msgr2=0x7fe564075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe55000e9c0 tx=0x7fe55000ecd0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 shutdown_connections 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe54c0a3ed0 0x7fe54c0a6390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564108a80 0x7fe5640757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564102a90 0x7fe564074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 --2- 172.21.15.63:0/2414883335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe564103450 0x7fe564075280 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 >> 172.21.15.63:0/2414883335 conn(0x7fe564079bf0 msgr2=0x7fe5641016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:18.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 shutdown_connections 2024-04-25T18:47:18.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:18.846+0000 7fe56c600700 1 -- 172.21.15.63:0/2414883335 wait complete. 2024-04-25T18:47:19.903 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-04-25T18:47:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:20 smithi070 ceph-mon[32820]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:20 smithi114 ceph-mon[32988]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:20 smithi063 ceph-mon[28006]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-25T18:47:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:22 smithi070 ceph-mon[32820]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T18:47:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:22 smithi114 ceph-mon[32988]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T18:47:22.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:22 smithi063 ceph-mon[28006]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T18:47:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:24 smithi070 ceph-mon[32820]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:24.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:24 smithi114 ceph-mon[32988]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:24 smithi063 ceph-mon[28006]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:26 smithi070 ceph-mon[32820]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-25T18:47:26.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:26 smithi114 ceph-mon[32988]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-25T18:47:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:26 smithi063 ceph-mon[28006]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-04-25T18:47:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:28 smithi070 ceph-mon[32820]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:28.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 -- 172.21.15.63:0/3338158861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e010e610 msgr2=0x7fd6e0110a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3338158861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e010e610 0x7fd6e0110a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009ca0 tx=0x7fd6d802a950 comp rx=0 tx=0).stop 2024-04-25T18:47:28.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 -- 172.21.15.63:0/3338158861 shutdown_connections 2024-04-25T18:47:28.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3338158861 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074c40 0x7fd6e010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3338158861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e010e610 0x7fd6e0110a20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3338158861 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e0074320 0x7fd6e0074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 -- 172.21.15.63:0/3338158861 >> 172.21.15.63:0/3338158861 conn(0x7fd6e00fd8a0 msgr2=0x7fd6e00ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 -- 172.21.15.63:0/3338158861 shutdown_connections 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.734+0000 7fd6e5800640 1 -- 172.21.15.63:0/3338158861 wait complete. 2024-04-25T18:47:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.735+0000 7fd6e5800640 1 Processor -- start 2024-04-25T18:47:28.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 -- start start 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e0074c40 0x7fd6e01a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e010e610 0x7fd6e019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e0114250 con 0x7fd6e0074c40 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01140d0 con 0x7fd6e0074320 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.736+0000 7fd6e5800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01143d0 con 0x7fd6e010e610 2024-04-25T18:47:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60070/0 (socket says 172.21.15.63:60070) 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 -- 172.21.15.63:0/3038416385 learned_addr learned my addr 172.21.15.63:0/3038416385 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6df400640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e0074c40 0x7fd6e01a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e010e610 0x7fd6e019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 -- 172.21.15.63:0/3038416385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e010e610 msgr2=0x7fd6e019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e010e610 0x7fd6e019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 -- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e0074c40 msgr2=0x7fd6e01a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e0074c40 0x7fd6e01a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.737+0000 7fd6dfe00640 1 -- 172.21.15.63:0/3038416385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e019adb0 con 0x7fd6e0074320 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000e9f0 tx=0x7fd6d000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6dcc00640 1 -- 172.21.15.63:0/3038416385 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d000cd80 con 0x7fd6e0074320 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6dcc00640 1 -- 172.21.15.63:0/3038416385 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d00054d0 con 0x7fd6e0074320 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6dcc00640 1 -- 172.21.15.63:0/3038416385 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0010690 con 0x7fd6e0074320 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e019b0a0 con 0x7fd6e0074320 2024-04-25T18:47:28.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.738+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e01a72f0 con 0x7fd6e0074320 2024-04-25T18:47:28.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 Processor -- start 2024-04-25T18:47:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 -- start start 2024-04-25T18:47:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 0x5562a7489650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a748bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562a73e4900 con 0x5562a747e9b0 2024-04-25T18:47:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562a738aef0 con 0x5562a747f2d0 2024-04-25T18:47:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.743+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562a73e08c0 con 0x5562a7489b90 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.744+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.744+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33380/0 (socket says 172.21.15.63:33380) 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.744+0000 7fd6e4e00640 1 -- 172.21.15.63:0/732180926 learned_addr learned my addr 172.21.15.63:0/732180926 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.744+0000 7fd6df400640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a748bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.744+0000 7fd6dcc00640 1 -- 172.21.15.63:0/3038416385 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fd6d0004e10 con 0x7fd6e0074320 2024-04-25T18:47:28.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 0x5562a7489650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6e4e00640 1 -- 172.21.15.63:0/732180926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 msgr2=0x5562a748bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a748bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6e4e00640 1 -- 172.21.15.63:0/732180926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 msgr2=0x5562a7489650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 0x5562a7489650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6e4e00640 1 -- 172.21.15.63:0/732180926 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a70f7210 con 0x5562a747e9b0 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 0x5562a7489650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:28.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.745+0000 7fd6df400640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a748bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.746+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0075270 tx=0x7fd6d8005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5de3fc74ce44de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.746+0000 7fd6d5600640 1 -- 172.21.15.63:0/732180926 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8048070 con 0x5562a747e9b0 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6d5600640 1 -- 172.21.15.63:0/732180926 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d8003e20 con 0x5562a747e9b0 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6d5600640 1 -- 172.21.15.63:0/732180926 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8034040 con 0x5562a747e9b0 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6dcc00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 0x7fd6c40a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6dcc00640 1 -- 172.21.15.63:0/3038416385 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fd6d00d0f30 con 0x7fd6e0074320 2024-04-25T18:47:28.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 0x7fd6c40a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/732180926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 msgr2=0x5562a747ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.747+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0075270 tx=0x7fd6d8005f80 comp rx=0 tx=0).stop 2024-04-25T18:47:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/732180926 shutdown_connections 2024-04-25T18:47:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747f2d0 0x5562a7489650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a748bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/732180926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a747ed90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/732180926 >> 172.21.15.63:0/732180926 conn(0x5562a747aec0 msgr2=0x5562a747b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/732180926 shutdown_connections 2024-04-25T18:47:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6dfe00640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 0x7fd6c40a6370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ce4f0 tx=0x7fd6d0004a90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.748+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/732180926 wait complete. 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.749+0000 7fd6e9bd0740 1 -- start start 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.749+0000 7fd6e9bd0740 1 Processor -- start 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 0x5562a74870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a74875e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562a748c760 con 0x5562a747f2d0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562a739e310 con 0x5562a747e9b0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562a748c8e0 con 0x5562a7489b90 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e9bd0740 10 client.0 _ll_register_callbacks cb 0x5562a736ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60098/0 (socket says 172.21.15.63:60098) 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e4e00640 1 -- 172.21.15.63:0/2221143372 learned_addr learned my addr 172.21.15.63:0/2221143372 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a74875e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.750+0000 7fd6df400640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 0x5562a74870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 -- 172.21.15.63:0/2221143372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 msgr2=0x5562a74875e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a74875e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 -- 172.21.15.63:0/2221143372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 msgr2=0x5562a74870a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 0x5562a74870a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 -- 172.21.15.63:0/2221143372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a71c3620 con 0x5562a747e9b0 2024-04-25T18:47:28.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6df400640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 0x5562a74870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8004af0 tx=0x7fd6d802fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6692e44051446bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.751+0000 7fd6ca000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6d804f980 con 0x5562a747e9b0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6d8003e20 con 0x5562a747e9b0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fd6d804fb20 con 0x5562a747e9b0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd6d804fd20 con 0x5562a747e9b0 2024-04-25T18:47:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6e9bd0740 10 client.25301 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.752+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562a7498a10 con 0x5562a747e9b0 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6d8003e20 con 0x5562a747e9b0 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd6d8030440 con 0x5562a747e9b0 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 10 client.25301 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 10 client.25301 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 20 client.25301 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a7499d60 0x5562a749c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 4 client.25301 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562a73f88b0 con 0x5562a7499d60 2024-04-25T18:47:28.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.753+0000 7fd6df400640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a7499d60 0x5562a749c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.754+0000 7fd6df400640 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a7499d60 0x5562a749c150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f268873b92f583 server_cookie=12ec849e003b84bc in_seq=0 out_seq=0 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.754+0000 7fd6ca000640 10 client.25301 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6ca000640 1 -- 172.21.15.63:0/2221143372 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562a73f88b0 con 0x5562a7499d60 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6ca000640 10 client.25301 handle_command_reply: tid=0 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 client.25301 shutdown 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 msgr2=0x5562a7486b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8004af0 tx=0x7fd6d802fb40 comp rx=0 tx=0).stop 2024-04-25T18:47:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 shutdown_connections 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a7499d60 0x5562a749c150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a7489b90 0x5562a74875e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747f2d0 0x5562a74870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2221143372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a747e9b0 0x5562a7486b60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 >> 172.21.15.63:0/2221143372 conn(0x5562a747aec0 msgr2=0x5562a7480c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.755+0000 7fd6ca000640 0 client.25301 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.756+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 shutdown_connections 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.756+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2221143372 wait complete. 2024-04-25T18:47:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.756+0000 7fd6e9bd0740 20 client.25301 trim_cache size 0 max 16384 2024-04-25T18:47:28.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.787+0000 7fd6e9bd0740 1 Processor -- start 2024-04-25T18:47:28.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.787+0000 7fd6e9bd0740 1 -- start start 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.787+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.787+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a7479790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01ae1c0 con 0x5562a747e9b0 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01aba90 con 0x5562a7489b90 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01a98e0 con 0x5562a747f2d0 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6df400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39448/0 (socket says 172.21.15.63:39448) 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6df400640 1 -- 172.21.15.63:0/4192285990 learned_addr learned my addr 172.21.15.63:0/4192285990 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33404/0 (socket says 172.21.15.63:33404) 2024-04-25T18:47:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.788+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a7479790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 -- 172.21.15.63:0/4192285990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 msgr2=0x5562a7479790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a7479790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 -- 172.21.15.63:0/4192285990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 msgr2=0x5562a749f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 -- 172.21.15.63:0/4192285990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562a74a1f90 con 0x5562a747f2d0 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6df400640 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc012580 tx=0x7fd6cc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ac97d1ad97e117 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6d4c00640 1 -- 172.21.15.63:0/4192285990 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc028070 con 0x5562a747f2d0 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6d4c00640 1 -- 172.21.15.63:0/4192285990 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6cc00f7f0 con 0x5562a747f2d0 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.789+0000 7fd6d4c00640 1 -- 172.21.15.63:0/4192285990 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc0044e0 con 0x5562a747f2d0 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/4192285990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 msgr2=0x5562a749f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc012580 tx=0x7fd6cc005f90 comp rx=0 tx=0).stop 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/4192285990 shutdown_connections 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a7479790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a749f6b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/4192285990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a749f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.790+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/4192285990 >> 172.21.15.63:0/4192285990 conn(0x5562a747aec0 msgr2=0x5562a7481f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.791+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/4192285990 shutdown_connections 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.791+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/4192285990 wait complete. 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.791+0000 7fd6e9bd0740 1 -- start start 2024-04-25T18:47:28.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.791+0000 7fd6e9bd0740 1 Processor -- start 2024-04-25T18:47:28.798 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a74f8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.798 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a74f8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.798 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a749e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.798 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01a9f50 con 0x5562a747e9b0 2024-04-25T18:47:28.798 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01ab3c0 con 0x5562a7489b90 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e01aa620 con 0x5562a747f2d0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e9bd0740 10 client.0 _ll_register_callbacks cb 0x7fd6d802f8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.792+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a74f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6e4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a74f8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33414/0 (socket says 172.21.15.63:33414) 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6e4e00640 1 -- 172.21.15.63:0/2430102610 learned_addr learned my addr 172.21.15.63:0/2430102610 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a74f8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a749e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 -- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 msgr2=0x5562a749e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a749e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 -- 172.21.15.63:0/2430102610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 msgr2=0x5562a74f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a74f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 -- 172.21.15.63:0/2430102610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562a749ef60 con 0x5562a747f2d0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6df400640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a74f8ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009460 tx=0x7fd6cc009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10186859a4744888 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:28.799 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.793+0000 7fd6ca000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc028040 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6cc00f7f0 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fd6cc004220 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd6cc004420 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6e9bd0740 10 client.25330 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5562a74ffe90 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6cc00f7f0 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.794+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fd6cc02b070 con 0x5562a747f2d0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 10 client.25330 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 10 client.25330 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 20 client.25330 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a75005b0 0x5562a75029d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 4 client.25330 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562a73f2320 con 0x5562a75005b0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a75005b0 0x5562a75029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:28.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6e4e00640 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a75005b0 0x5562a75029d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8dd6ef51fcc2c39 server_cookie=ff30f37d50c372ad in_seq=0 out_seq=0 2024-04-25T18:47:28.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.795+0000 7fd6ca000640 10 client.25330 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6ca000640 1 -- 172.21.15.63:0/2430102610 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562a73f2320 con 0x5562a75005b0 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6ca000640 10 client.25330 handle_command_reply: tid=0 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 client.25330 shutdown 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 msgr2=0x5562a74f8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a74f8ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc009460 tx=0x7fd6cc009e50 comp rx=0 tx=0).stop 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 shutdown_connections 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562a75005b0 0x5562a75029d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562a7489b90 0x5562a749e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562a747f2d0 0x5562a74f8ba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 --2- 172.21.15.63:0/2430102610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562a747e9b0 0x5562a74f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 >> 172.21.15.63:0/2430102610 conn(0x5562a747aec0 msgr2=0x5562a7479790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.801+0000 7fd6ca000640 0 client.25330 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.802+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 shutdown_connections 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.802+0000 7fd6e9bd0740 1 -- 172.21.15.63:0/2430102610 wait complete. 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.802+0000 7fd6e9bd0740 20 client.25330 trim_cache size 0 max 16384 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:47:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 msgr2=0x7fd6c40a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 0x7fd6c40a6370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00ce4f0 tx=0x7fd6d0004a90 comp rx=0 tx=0).stop 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 msgr2=0x7fd6e01a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000e9f0 tx=0x7fd6d000edf0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 shutdown_connections 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6c40a3eb0 0x7fd6c40a6370 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd6e010e610 0x7fd6e019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6e0074c40 0x7fd6e01a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 --2- 172.21.15.63:0/3038416385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6e0074320 0x7fd6e01a0430 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 >> 172.21.15.63:0/3038416385 conn(0x7fd6e00fd8a0 msgr2=0x7fd6e00fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 shutdown_connections 2024-04-25T18:47:28.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:28.803+0000 7fd6e5800640 1 -- 172.21.15.63:0/3038416385 wait complete. 2024-04-25T18:47:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:28 smithi114 ceph-mon[32988]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:28 smithi063 ceph-mon[28006]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:30 smithi070 ceph-mon[32820]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:30 smithi114 ceph-mon[32988]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:30.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:30 smithi063 ceph-mon[28006]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-04-25T18:47:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:32 smithi070 ceph-mon[32820]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T18:47:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:32 smithi114 ceph-mon[32988]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T18:47:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:32 smithi063 ceph-mon[28006]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T18:47:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:34 smithi070 ceph-mon[32820]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:34 smithi114 ceph-mon[32988]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:34.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:34 smithi063 ceph-mon[28006]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:34.904 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 10 2024-04-25T18:47:34.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:47:35.360 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:36 smithi063 ceph-mon[28006]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T18:47:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:36 smithi114 ceph-mon[32988]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T18:47:36.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:36 smithi070 ceph-mon[32820]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 8 op/s 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 -- 172.21.15.63:0/19275541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 msgr2=0x7f078406abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 --2- 172.21.15.63:0/19275541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078406abb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0770009990 tx=0x7f0770009ca0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 -- 172.21.15.63:0/19275541 shutdown_connections 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 --2- 172.21.15.63:0/19275541 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0784108a30 0x7f0784108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 --2- 172.21.15.63:0/19275541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f078406b0f0 0x7f078410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 --2- 172.21.15.63:0/19275541 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078406abb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 -- 172.21.15.63:0/19275541 >> 172.21.15.63:0/19275541 conn(0x7f07840788f0 msgr2=0x7f0784078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:37.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 -- 172.21.15.63:0/19275541 shutdown_connections 2024-04-25T18:47:37.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.730+0000 7f0788e00700 1 -- 172.21.15.63:0/19275541 wait complete. 2024-04-25T18:47:37.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.732+0000 7f0788e00700 1 Processor -- start 2024-04-25T18:47:37.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.732+0000 7f0788e00700 1 -- start start 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f078406b0f0 0x7f078419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 0x7f078419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0784112bd0 con 0x7f0784108a30 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0784112a50 con 0x7f078406a750 2024-04-25T18:47:37.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0788e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0784112d50 con 0x7f078406b0f0 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0781e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0781e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56994/0 (socket says 172.21.15.63:56994) 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0781e00700 1 -- 172.21.15.63:0/1743188405 learned_addr learned my addr 172.21.15.63:0/1743188405 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.733+0000 7f0782800700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 0x7f078419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781400700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f078406b0f0 0x7f078419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 -- 172.21.15.63:0/1743188405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f078406b0f0 msgr2=0x7f078419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:37.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f078406b0f0 0x7f078419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 -- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 msgr2=0x7f078419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 0x7f078419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 -- 172.21.15.63:0/1743188405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07841a0330 con 0x7f078406a750 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0782800700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 0x7f078419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.734+0000 7f0781e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f077400e9a0 tx=0x7f077400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.735+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077400cbd0 con 0x7f078406a750 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.735+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f077400cd70 con 0x7f078406a750 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.735+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774018710 con 0x7f078406a750 2024-04-25T18:47:37.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.735+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07841a0620 con 0x7f078406a750 2024-04-25T18:47:37.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.735+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07841a7ea0 con 0x7f078406a750 2024-04-25T18:47:37.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.737+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0784104170 con 0x7f078406a750 2024-04-25T18:47:37.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.740+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f0774010a00 con 0x7f078406a750 2024-04-25T18:47:37.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.742+0000 7f077aa00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 0x7f07680a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:37.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.742+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f07740d0f10 con 0x7f078406a750 2024-04-25T18:47:37.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.743+0000 7f0781400700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 0x7f07680a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:37.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.744+0000 7f0781400700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 0x7f07680a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f077000b2e0 tx=0x7f0770005e80 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:37.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:37.746+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0774053ff0 con 0x7f078406a750 2024-04-25T18:47:38.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.215+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078419c6b0 con 0x7f078406a750 2024-04-25T18:47:38.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.216+0000 7f077aa00700 1 -- 172.21.15.63:0/1743188405 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11868 (secure 0 0 0) 0x7f078419c6b0 con 0x7f078406a750 2024-04-25T18:47:38.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:47:38.219 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:46:32.730913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:47:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.219+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 msgr2=0x7f07680a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.219+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 0x7f07680a6410 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f077000b2e0 tx=0x7f0770005e80 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.219+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 msgr2=0x7f078419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.219+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f077400e9a0 tx=0x7f077400ed60 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 shutdown_connections 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f07680a3f50 0x7f07680a6410 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0784108a30 0x7f078419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078406a750 0x7f078419b320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 --2- 172.21.15.63:0/1743188405 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f078406b0f0 0x7f078419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 >> 172.21.15.63:0/1743188405 conn(0x7f07840788f0 msgr2=0x7f078410df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:38.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 shutdown_connections 2024-04-25T18:47:38.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:38.220+0000 7f0788e00700 1 -- 172.21.15.63:0/1743188405 wait complete. 2024-04-25T18:47:38.224 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-04-25T18:47:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:38 smithi063 ceph-mon[28006]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:38 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1743188405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:38 smithi114 ceph-mon[32988]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:38 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1743188405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:38 smithi070 ceph-mon[32820]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:38 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1743188405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:39.098 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 70 --await-for 42.0' 2024-04-25T18:47:39.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 70 --await-for 42.0 2024-04-25T18:47:39.549 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:40.725 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:40 smithi063 ceph-mon[28006]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:40 smithi114 ceph-mon[32988]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:40 smithi070 ceph-mon[32820]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-25T18:47:41.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 -- 172.21.15.63:0/4293203048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 msgr2=0x7fa660113380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 --2- 172.21.15.63:0/4293203048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa660113380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009990 tx=0x7fa64c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 -- 172.21.15.63:0/4293203048 shutdown_connections 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 --2- 172.21.15.63:0/4293203048 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa660074e00 0x7fa6600751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 --2- 172.21.15.63:0/4293203048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6601138c0 0x7fa660115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 --2- 172.21.15.63:0/4293203048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa660113380 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.835+0000 7fa65f200700 1 -- 172.21.15.63:0/4293203048 >> 172.21.15.63:0/4293203048 conn(0x7fa6600fe3a0 msgr2=0x7fa6601007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.836+0000 7fa65f200700 1 -- 172.21.15.63:0/4293203048 shutdown_connections 2024-04-25T18:47:41.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.836+0000 7fa65f200700 1 -- 172.21.15.63:0/4293203048 wait complete. 2024-04-25T18:47:41.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.837+0000 7fa65f200700 1 Processor -- start 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.837+0000 7fa65f200700 1 -- start start 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.837+0000 7fa65f200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.837+0000 7fa65f200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa66019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65f200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6601138c0 0x7fa6601a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65f200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6601191d0 con 0x7fa660074e00 2024-04-25T18:47:41.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65f200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa660119050 con 0x7fa660075720 2024-04-25T18:47:41.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65f200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa660119350 con 0x7fa6601138c0 2024-04-25T18:47:41.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65de00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40048/0 (socket says 172.21.15.63:40048) 2024-04-25T18:47:41.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65de00700 1 -- 172.21.15.63:0/4292087247 learned_addr learned my addr 172.21.15.63:0/4292087247 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.838+0000 7fa65d400700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa66019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65e800700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6601138c0 0x7fa6601a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 -- 172.21.15.63:0/4292087247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6601138c0 msgr2=0x7fa6601a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6601138c0 0x7fa6601a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 -- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 msgr2=0x7fa66019f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 -- 172.21.15.63:0/4292087247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6601a45f0 con 0x7fa660075720 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65de00700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.839+0000 7fa65d400700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa66019fb20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0098e0 tx=0x7fa64c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.840+0000 7fa656a00700 1 -- 172.21.15.63:0/4292087247 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c027070 con 0x7fa660075720 2024-04-25T18:47:41.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.840+0000 7fa656a00700 1 -- 172.21.15.63:0/4292087247 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa64c00b610 con 0x7fa660075720 2024-04-25T18:47:41.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.840+0000 7fa656a00700 1 -- 172.21.15.63:0/4292087247 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0173f0 con 0x7fa660075720 2024-04-25T18:47:41.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.840+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6601a4880 con 0x7fa660075720 2024-04-25T18:47:41.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.840+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66006c460 con 0x7fa660075720 2024-04-25T18:47:41.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.845+0000 7fa656a00700 1 -- 172.21.15.63:0/4292087247 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fa64c02a070 con 0x7fa660075720 2024-04-25T18:47:41.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.846+0000 7fa66fa31b80 1 Processor -- start 2024-04-25T18:47:41.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.846+0000 7fa66fa31b80 1 -- start start 2024-04-25T18:47:41.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559975ffa020 con 0x5599760a7850 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559975fa08c0 con 0x5599760a6f30 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559975ff5fe0 con 0x5599760b2700 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.847+0000 7fa656a00700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 0x7fa6440a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa656a00700 1 -- 172.21.15.63:0/4292087247 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa64c0d1a00 con 0x7fa660075720 2024-04-25T18:47:41.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57018/0 (socket says 172.21.15.63:57018) 2024-04-25T18:47:41.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 -- 172.21.15.63:0/3415014992 learned_addr learned my addr 172.21.15.63:0/3415014992 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65d400700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65de00700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 -- 172.21.15.63:0/3415014992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 msgr2=0x5599760b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 -- 172.21.15.63:0/3415014992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 msgr2=0x5599760b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.848+0000 7fa65e800700 1 -- 172.21.15.63:0/3415014992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559975f96810 con 0x5599760a6f30 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.849+0000 7fa65de00700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.849+0000 7fa65e800700 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa65800f6b0 tx=0x7fa65800f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb6e84419ba38310 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.849+0000 7fa64b400700 1 -- 172.21.15.63:0/3415014992 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65801b030 con 0x5599760a6f30 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.849+0000 7fa64b400700 1 -- 172.21.15.63:0/3415014992 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6580092e0 con 0x5599760a6f30 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.849+0000 7fa64b400700 1 -- 172.21.15.63:0/3415014992 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658007500 con 0x5599760a6f30 2024-04-25T18:47:41.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.850+0000 7fa65e800700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 0x7fa6440a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.850+0000 7fa66fa31b80 1 -- 172.21.15.63:0/3415014992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 msgr2=0x5599760a7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.850+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa65800f6b0 tx=0x7fa65800f9c0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa65e800700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 0x7fa6440a64b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa658007730 tx=0x7fa6580076a0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa66fa31b80 1 -- 172.21.15.63:0/3415014992 shutdown_connections 2024-04-25T18:47:41.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760a7310 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/3415014992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.851+0000 7fa66fa31b80 1 -- 172.21.15.63:0/3415014992 >> 172.21.15.63:0/3415014992 conn(0x559975fd60e0 msgr2=0x5599760a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.852+0000 7fa66fa31b80 1 -- 172.21.15.63:0/3415014992 shutdown_connections 2024-04-25T18:47:41.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.852+0000 7fa66fa31b80 1 -- 172.21.15.63:0/3415014992 wait complete. 2024-04-25T18:47:41.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.853+0000 7fa66fa31b80 1 -- start start 2024-04-25T18:47:41.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.853+0000 7fa66fa31b80 1 Processor -- start 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5599760b5190 con 0x5599760a7850 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559975fb39f0 con 0x5599760a6f30 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5599760b5310 con 0x5599760b2700 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.854+0000 7fa66fa31b80 10 client.0 _ll_register_callbacks cb 0x559975f928b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57032/0 (socket says 172.21.15.63:57032) 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65de00700 1 -- 172.21.15.63:0/518281573 learned_addr learned my addr 172.21.15.63:0/518281573 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65e800700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65de00700 1 -- 172.21.15.63:0/518281573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 msgr2=0x5599760af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.855+0000 7fa65de00700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa65de00700 1 -- 172.21.15.63:0/518281573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 msgr2=0x5599760af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa65de00700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa65de00700 1 -- 172.21.15.63:0/518281573 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559975f47a60 con 0x5599760a6f30 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa65de00700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa650011ec0 tx=0x7fa65000f640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=574382bac56adb2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa63fe00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65001f060 con 0x5599760a6f30 2024-04-25T18:47:41.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.856+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa650012450 con 0x5599760a6f30 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.857+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6580092e0 con 0x5599760a6f30 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.857+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa65000fe50 con 0x5599760a6f30 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.857+0000 7fa66fa31b80 10 client.25325 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.857+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x559975f90c80 con 0x5599760a6f30 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.858+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6580092e0 con 0x5599760a6f30 2024-04-25T18:47:41.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.858+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa65001a7c0 con 0x5599760a6f30 2024-04-25T18:47:41.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 10 client.25325 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:41.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 10 client.25325 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:47:41.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 20 client.25325 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5599760c1dc0 0x5599760c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 4 client.25325 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559976007810 con 0x5599760c1dc0 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.859+0000 7fa65d400700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5599760c1dc0 0x5599760c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.860+0000 7fa65d400700 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5599760c1dc0 0x5599760c41b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bbd7805f931286f server_cookie=57f080a1fd5975a1 in_seq=0 out_seq=0 2024-04-25T18:47:41.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.860+0000 7fa63fe00700 10 client.25325 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.861+0000 7fa63fe00700 1 -- 172.21.15.63:0/518281573 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559976007810 con 0x5599760c1dc0 2024-04-25T18:47:41.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.861+0000 7fa63fe00700 10 client.25325 handle_command_reply: tid=0 2024-04-25T18:47:41.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.861+0000 7fa66fa31b80 1 client.25325 shutdown 2024-04-25T18:47:41.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 msgr2=0x5599760b1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa650011ec0 tx=0x7fa65000f640 comp rx=0 tx=0).stop 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 shutdown_connections 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5599760c1dc0 0x5599760c41b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a6f30 0x5599760b1eb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/518281573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760a7850 0x5599760af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 >> 172.21.15.63:0/518281573 conn(0x559975fd60e0 msgr2=0x5599760a3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa63fe00700 0 client.25325 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.862+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 shutdown_connections 2024-04-25T18:47:41.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.863+0000 7fa66fa31b80 1 -- 172.21.15.63:0/518281573 wait complete. 2024-04-25T18:47:41.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.863+0000 7fa66fa31b80 20 client.25325 trim_cache size 0 max 16384 2024-04-25T18:47:41.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.903+0000 7fa66fa31b80 1 Processor -- start 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.903+0000 7fa66fa31b80 1 -- start start 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760be050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760ba560 0x5599760ba9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6601be730 con 0x5599760ba560 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa660073ab0 con 0x5599760a7850 2024-04-25T18:47:41.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa66fa31b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa660071920 con 0x5599760b2700 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa65de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa65de00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57046/0 (socket says 172.21.15.63:57046) 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa65de00700 1 -- 172.21.15.63:0/951771174 learned_addr learned my addr 172.21.15.63:0/951771174 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.904+0000 7fa65d400700 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760be050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65e800700 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760ba560 0x5599760ba9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 -- 172.21.15.63:0/951771174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 msgr2=0x5599760be050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760be050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 -- 172.21.15.63:0/951771174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760ba560 msgr2=0x5599760ba9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760ba560 0x5599760ba9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 -- 172.21.15.63:0/951771174 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599760c4810 con 0x5599760a7850 2024-04-25T18:47:41.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.905+0000 7fa65de00700 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa650011e90 tx=0x7fa650005eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e2ddf21928a994 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.906+0000 7fa64aa00700 1 -- 172.21.15.63:0/951771174 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650029080 con 0x5599760a7850 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.906+0000 7fa64aa00700 1 -- 172.21.15.63:0/951771174 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa65000ab40 con 0x5599760a7850 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 -- 172.21.15.63:0/951771174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 msgr2=0x5599760af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa650011e90 tx=0x7fa650005eb0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 -- 172.21.15.63:0/951771174 shutdown_connections 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760ba560 0x5599760ba9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760a7850 0x5599760af7e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/951771174 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760b2700 0x5599760be050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.907+0000 7fa66fa31b80 1 -- 172.21.15.63:0/951771174 >> 172.21.15.63:0/951771174 conn(0x559975fd60e0 msgr2=0x5599760b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.908+0000 7fa66fa31b80 1 -- 172.21.15.63:0/951771174 shutdown_connections 2024-04-25T18:47:41.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.908+0000 7fa66fa31b80 1 -- 172.21.15.63:0/951771174 wait complete. 2024-04-25T18:47:41.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.908+0000 7fa66fa31b80 1 -- start start 2024-04-25T18:47:41.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.909+0000 7fa66fa31b80 1 Processor -- start 2024-04-25T18:47:41.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.909+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 0x5599760b9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa65de00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa65de00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39846/0 (socket says 172.21.15.63:39846) 2024-04-25T18:47:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa66fa31b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760ba560 0x559976121560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa65de00700 1 -- 172.21.15.63:0/161070696 learned_addr learned my addr 172.21.15.63:0/161070696 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa660071f70 con 0x5599760b2700 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6600733e0 con 0x5599760ba560 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa660072640 con 0x5599760a7850 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.910+0000 7fa65e800700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760ba560 0x559976121560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa66fa31b80 10 client.0 _ll_register_callbacks cb 0x559975f928b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65d400700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 0x5599760b9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65de00700 1 -- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760ba560 msgr2=0x559976121560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65de00700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760ba560 0x559976121560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65de00700 1 -- 172.21.15.63:0/161070696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 msgr2=0x5599760b9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65de00700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 0x5599760b9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65de00700 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559976121c40 con 0x5599760a7850 2024-04-25T18:47:41.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.911+0000 7fa65d400700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 0x5599760b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.912+0000 7fa65de00700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa65001acb0 tx=0x7fa650005b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb621ba27e838595 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.912+0000 7fa63fe00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.912+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65003c060 con 0x5599760a7850 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.912+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fa65001a710 con 0x5599760a7850 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.913+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65000ab40 con 0x5599760a7850 2024-04-25T18:47:41.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.913+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa65001f040 con 0x5599760a7850 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.913+0000 7fa66fa31b80 10 client.25348 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.913+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55997612a120 con 0x5599760a7850 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa65000ab40 con 0x5599760a7850 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fa650023780 con 0x5599760a7850 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 10 client.25348 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 10 client.25348 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 20 client.25348 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:41.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55997612a690 0x55997612cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 4 client.25348 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}] 2024-04-25T18:47:41.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.914+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 70.0, "await_for": 42.0}) -- 0x5599760018b0 con 0x55997612a690 2024-04-25T18:47:41.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.915+0000 7fa65d400700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55997612a690 0x55997612cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.915+0000 7fa63fe00700 10 client.25348 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.917+0000 7fa65d400700 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55997612a690 0x55997612cab0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4882f22ef7c95f90 server_cookie=98f9cc130ba51ef3 in_seq=0 out_seq=0 2024-04-25T18:47:41.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.926+0000 7fa63fe00700 1 -- 172.21.15.63:0/161070696 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x5599760018b0 con 0x55997612a690 2024-04-25T18:47:41.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.926+0000 7fa63fe00700 10 client.25348 handle_command_reply: tid=0 2024-04-25T18:47:41.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.926+0000 7fa66fa31b80 1 client.25348 shutdown 2024-04-25T18:47:41.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.926+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 msgr2=0x5599760b9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.926+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa65001acb0 tx=0x7fa650005b00 comp rx=0 tx=0).stop 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 shutdown_connections 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55997612a690 0x55997612cab0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5599760a7850 0x5599760b9230 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5599760ba560 0x559976121560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 --2- 172.21.15.63:0/161070696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5599760b2700 0x5599760b9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 >> 172.21.15.63:0/161070696 conn(0x559975fd60e0 msgr2=0x5599760b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa63fe00700 0 client.25348 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 shutdown_connections 2024-04-25T18:47:41.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.927+0000 7fa66fa31b80 1 -- 172.21.15.63:0/161070696 wait complete. 2024-04-25T18:47:41.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.928+0000 7fa66fa31b80 20 client.25348 trim_cache size 0 max 16384 2024-04-25T18:47:41.930 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 24, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 2, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "55d5471": { 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 2, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 70.0, 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:47:41.931 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:47:41.932 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 msgr2=0x7fa6440a64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 0x7fa6440a64b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa658007730 tx=0x7fa6580076a0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 msgr2=0x7fa66019fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa66019fb20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0098e0 tx=0x7fa64c0057c0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 shutdown_connections 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6440a3ff0 0x7fa6440a64b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6601138c0 0x7fa6601a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa660074e00 0x7fa66019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 --2- 172.21.15.63:0/4292087247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa660075720 0x7fa66019fb20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.929+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 >> 172.21.15.63:0/4292087247 conn(0x7fa6600fe3a0 msgr2=0x7fa6600ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.930+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 shutdown_connections 2024-04-25T18:47:41.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:41.930+0000 7fa65f200700 1 -- 172.21.15.63:0/4292087247 wait complete. 2024-04-25T18:47:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 -- 172.21.15.63:0/2711038372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff178104d50 msgr2=0x7ff178108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 --2- 172.21.15.63:0/2711038372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff178104d50 0x7ff178108d90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff164009900 tx=0x7ff16402a3f0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 -- 172.21.15.63:0/2711038372 shutdown_connections 2024-04-25T18:47:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 --2- 172.21.15.63:0/2711038372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff178111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 --2- 172.21.15.63:0/2711038372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff178104d50 0x7ff178108d90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 --2- 172.21.15.63:0/2711038372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 -- 172.21.15.63:0/2711038372 >> 172.21.15.63:0/2711038372 conn(0x7ff1780ffc40 msgr2=0x7ff178102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 -- 172.21.15.63:0/2711038372 shutdown_connections 2024-04-25T18:47:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.963+0000 7ff180600640 1 -- 172.21.15.63:0/2711038372 wait complete. 2024-04-25T18:47:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.965+0000 7ff180600640 1 Processor -- start 2024-04-25T18:47:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.965+0000 7ff180600640 1 -- start start 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.965+0000 7ff180600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178110f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.965+0000 7ff180600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff17810dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff180600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17810e4e0 0x7ff17810e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff180600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781143e0 con 0x7ff178109470 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff180600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff178114260 con 0x7ff1781043a0 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff180600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff178114560 con 0x7ff17810e4e0 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff17dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178110f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff17dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178110f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57072/0 (socket says 172.21.15.63:57072) 2024-04-25T18:47:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff17d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff17810dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff17dc00640 1 -- 172.21.15.63:0/4251338667 learned_addr learned my addr 172.21.15.63:0/4251338667 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.966+0000 7ff17e600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17810e4e0 0x7ff17810e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.967+0000 7ff17d200640 1 -- 172.21.15.63:0/4251338667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17810e4e0 msgr2=0x7ff17810e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.967+0000 7ff17d200640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17810e4e0 0x7ff17810e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.967+0000 7ff17d200640 1 -- 172.21.15.63:0/4251338667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 msgr2=0x7ff178110f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.967+0000 7ff17d200640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178110f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.967+0000 7ff17d200640 1 -- 172.21.15.63:0/4251338667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1781ab330 con 0x7ff178109470 2024-04-25T18:47:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.968+0000 7ff17d200640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff17810dfa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff16402a910 tx=0x7ff164003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.969+0000 7ff176000640 1 -- 172.21.15.63:0/4251338667 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16403f070 con 0x7ff178109470 2024-04-25T18:47:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.969+0000 7ff176000640 1 -- 172.21.15.63:0/4251338667 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff164031d60 con 0x7ff178109470 2024-04-25T18:47:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.969+0000 7ff176000640 1 -- 172.21.15.63:0/4251338667 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164035030 con 0x7ff178109470 2024-04-25T18:47:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.969+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1781ab5c0 con 0x7ff178109470 2024-04-25T18:47:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.970+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1781aba30 con 0x7ff178109470 2024-04-25T18:47:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.974+0000 7ff1833a4740 1 Processor -- start 2024-04-25T18:47:41.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.975+0000 7ff1833a4740 1 -- start start 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x565211825c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x5652118304c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x565211832df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56521178b900 con 0x565211830a00 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x565211731ef0 con 0x565211826140 2024-04-25T18:47:41.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff1833a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5652117878c0 con 0x565211825820 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.976+0000 7ff17e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x565211825c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.978+0000 7ff17e600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x565211825c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39876/0 (socket says 172.21.15.63:39876) 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.978+0000 7ff17e600640 1 -- 172.21.15.63:0/2616357097 learned_addr learned my addr 172.21.15.63:0/2616357097 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.978+0000 7ff17dc00640 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x5652118304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.978+0000 7ff176000640 1 -- 172.21.15.63:0/4251338667 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7ff16403bd10 con 0x7ff178109470 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.979+0000 7ff17dc00640 1 -- 172.21.15.63:0/2616357097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 msgr2=0x565211825c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.979+0000 7ff17dc00640 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x565211825c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.979+0000 7ff17dc00640 1 -- 172.21.15.63:0/2616357097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 msgr2=0x565211832df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.979+0000 7ff17d200640 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x565211832df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.980+0000 7ff17dc00640 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x565211832df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.980+0000 7ff17dc00640 1 -- 172.21.15.63:0/2616357097 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56521149e210 con 0x565211826140 2024-04-25T18:47:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.980+0000 7ff17dc00640 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x5652118304c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0058c0 tx=0x7ff16c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0211ff25cf643a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.981+0000 7ff16aa00640 1 -- 172.21.15.63:0/2616357097 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c004230 con 0x565211826140 2024-04-25T18:47:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.981+0000 7ff176000640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 0x7ff1600a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.981+0000 7ff16aa00640 1 -- 172.21.15.63:0/2616357097 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16c00be90 con 0x565211826140 2024-04-25T18:47:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.981+0000 7ff16aa00640 1 -- 172.21.15.63:0/2616357097 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0116c0 con 0x565211826140 2024-04-25T18:47:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.982+0000 7ff176000640 1 -- 172.21.15.63:0/4251338667 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff1640f13b0 con 0x7ff178109470 2024-04-25T18:47:41.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.982+0000 7ff17e600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 0x7ff1600a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.983+0000 7ff1833a4740 1 -- 172.21.15.63:0/2616357097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 msgr2=0x5652118304c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.983+0000 7ff1833a4740 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x5652118304c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0058c0 tx=0x7ff16c0107a0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.983+0000 7ff17e600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 0x7ff1600a64d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff170005f40 tx=0x7ff1700096a0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 -- 172.21.15.63:0/2616357097 shutdown_connections 2024-04-25T18:47:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x5652118304c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x565211832df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 --2- 172.21.15.63:0/2616357097 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x565211825c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 -- 172.21.15.63:0/2616357097 >> 172.21.15.63:0/2616357097 conn(0x565211821d30 msgr2=0x565211822110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 -- 172.21.15.63:0/2616357097 shutdown_connections 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.984+0000 7ff1833a4740 1 -- 172.21.15.63:0/2616357097 wait complete. 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.985+0000 7ff1833a4740 1 -- start start 2024-04-25T18:47:41.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.985+0000 7ff1833a4740 1 Processor -- start 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521182d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x565211833630 con 0x565211826140 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x565211745310 con 0x565211830a00 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5652118337b0 con 0x565211825820 2024-04-25T18:47:41.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff1833a4740 10 client.0 _ll_register_callbacks cb 0x565211713cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff17e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff17d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff17e600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57098/0 (socket says 172.21.15.63:57098) 2024-04-25T18:47:41.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.987+0000 7ff17d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40132/0 (socket says 172.21.15.63:40132) 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17e600640 1 -- 172.21.15.63:0/1551159375 learned_addr learned my addr 172.21.15.63:0/1551159375 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521182d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 -- 172.21.15.63:0/1551159375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 msgr2=0x56521182d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521182d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 -- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 msgr2=0x56521182e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 -- 172.21.15.63:0/1551159375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56521156a620 con 0x565211826140 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521182d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17e600640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff17d200640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff1640ee910 tx=0x7ff1640b9400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92022f8752f674fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:41.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.988+0000 7ff15f400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1640bcac0 con 0x565211826140 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff16c00be90 con 0x565211826140 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff1640c1160 con 0x565211826140 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff1640bd860 con 0x565211826140 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff1833a4740 10 client.15192 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:41.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.989+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x565211833630 con 0x565211826140 2024-04-25T18:47:41.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.990+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16c00be90 con 0x565211826140 2024-04-25T18:47:41.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff1640bd3c0 con 0x565211826140 2024-04-25T18:47:41.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 10 client.15192 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 10 client.15192 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 20 client.15192 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x565211840db0 0x5652118431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 4 client.15192 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56521179f8b0 con 0x565211840db0 2024-04-25T18:47:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.991+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x565211840db0 0x5652118431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.992+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x565211840db0 0x5652118431d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33d085c8aee8548 server_cookie=c88979d2d1174c89 in_seq=0 out_seq=0 2024-04-25T18:47:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.992+0000 7ff15f400640 10 client.15192 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.993+0000 7ff15f400640 1 -- 172.21.15.63:0/1551159375 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56521179f8b0 con 0x565211840db0 2024-04-25T18:47:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.993+0000 7ff15f400640 10 client.15192 handle_command_reply: tid=0 2024-04-25T18:47:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.993+0000 7ff1833a4740 1 client.15192 shutdown 2024-04-25T18:47:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 msgr2=0x56521182deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff1640ee910 tx=0x7ff1640b9400 comp rx=0 tx=0).stop 2024-04-25T18:47:41.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 shutdown_connections 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x565211840db0 0x5652118431d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x56521182e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211826140 0x56521182deb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1551159375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521182d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 >> 172.21.15.63:0/1551159375 conn(0x565211821d30 msgr2=0x565211827a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.994+0000 7ff15f400640 0 client.15192 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.995+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 shutdown_connections 2024-04-25T18:47:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.995+0000 7ff1833a4740 1 -- 172.21.15.63:0/1551159375 wait complete. 2024-04-25T18:47:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:41.995+0000 7ff1833a4740 20 client.15192 trim_cache size 0 max 16384 2024-04-25T18:47:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.027+0000 7ff1833a4740 1 Processor -- start 2024-04-25T18:47:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.027+0000 7ff1833a4740 1 -- start start 2024-04-25T18:47:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.027+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.027+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x565211821610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x5652118278e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff1833a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b6a10 con 0x565211830a00 2024-04-25T18:47:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff1833a4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b42e0 con 0x565211826140 2024-04-25T18:47:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff1833a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b2130 con 0x565211825820 2024-04-25T18:47:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff17dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff17dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39906/0 (socket says 172.21.15.63:39906) 2024-04-25T18:47:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff17dc00640 1 -- 172.21.15.63:0/1694556662 learned_addr learned my addr 172.21.15.63:0/1694556662 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff17d200640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x565211821610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.028+0000 7ff17e600640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x5652118278e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 -- 172.21.15.63:0/1694556662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 msgr2=0x565211821610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x565211821610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 -- 172.21.15.63:0/1694556662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 msgr2=0x5652118278e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x5652118278e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 -- 172.21.15.63:0/1694556662 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565211825f70 con 0x565211825820 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17d200640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x565211821610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17e600640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x5652118278e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff17dc00640 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006010 tx=0x7ff16c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2852ab7075a0ccae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff16a000640 1 -- 172.21.15.63:0/1694556662 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c02f070 con 0x565211825820 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff16a000640 1 -- 172.21.15.63:0/1694556662 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16c005070 con 0x565211825820 2024-04-25T18:47:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.029+0000 7ff16a000640 1 -- 172.21.15.63:0/1694556662 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c02a2d0 con 0x565211825820 2024-04-25T18:47:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 -- 172.21.15.63:0/1694556662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 msgr2=0x56521183c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006010 tx=0x7ff16c013040 comp rx=0 tx=0).stop 2024-04-25T18:47:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 -- 172.21.15.63:0/1694556662 shutdown_connections 2024-04-25T18:47:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211826140 0x565211821610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211830a00 0x5652118278e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 --2- 172.21.15.63:0/1694556662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211825820 0x56521183c4f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.030+0000 7ff1833a4740 1 -- 172.21.15.63:0/1694556662 >> 172.21.15.63:0/1694556662 conn(0x565211821d30 msgr2=0x5652118271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.031+0000 7ff1833a4740 1 -- 172.21.15.63:0/1694556662 shutdown_connections 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.031+0000 7ff1833a4740 1 -- 172.21.15.63:0/1694556662 wait complete. 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.031+0000 7ff1833a4740 1 -- start start 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.031+0000 7ff1833a4740 1 Processor -- start 2024-04-25T18:47:42.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 0x565211838df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x5652118366c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b27a0 con 0x565211825820 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b3c10 con 0x565211830a00 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff1781b2e70 con 0x565211826140 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.032+0000 7ff1833a4740 10 client.0 _ll_register_callbacks cb 0x7ff1640b8ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.033+0000 7ff17d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.033+0000 7ff17d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39914/0 (socket says 172.21.15.63:39914) 2024-04-25T18:47:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.033+0000 7ff17d200640 1 -- 172.21.15.63:0/647324703 learned_addr learned my addr 172.21.15.63:0/647324703 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17e600640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x5652118366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17dc00640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 0x565211838df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17d200640 1 -- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 msgr2=0x5652118366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17d200640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x5652118366c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17d200640 1 -- 172.21.15.63:0/647324703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 msgr2=0x565211838df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17d200640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 0x565211838df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17d200640 1 -- 172.21.15.63:0/647324703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565211836ec0 con 0x565211826140 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.034+0000 7ff17dc00640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 0x565211838df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff17d200640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff164000c00 tx=0x7ff1640b9a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e4cca266da68023 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff15f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1640bcac0 con 0x565211826140 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff1640318b0 con 0x565211826140 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff16c005070 con 0x565211826140 2024-04-25T18:47:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff1640c1510 con 0x565211826140 2024-04-25T18:47:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff1833a4740 10 client.25366 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.035+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5652118a72f0 con 0x565211826140 2024-04-25T18:47:42.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.036+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16c005070 con 0x565211826140 2024-04-25T18:47:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7ff1640bc620 con 0x565211826140 2024-04-25T18:47:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 10 client.25366 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 10 client.25366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 20 client.25366 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:42.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5652118a7800 0x5652118a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 4 client.25366 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:47:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.037+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565211799320 con 0x5652118a7800 2024-04-25T18:47:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.038+0000 7ff17dc00640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5652118a7800 0x5652118a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:42.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.038+0000 7ff17dc00640 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5652118a7800 0x5652118a9c20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5ddf26a25371861 server_cookie=9af870626d554e7a in_seq=0 out_seq=0 2024-04-25T18:47:42.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.038+0000 7ff15f400640 10 client.25366 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:42.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff15f400640 1 -- 172.21.15.63:0/647324703 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565211799320 con 0x5652118a7800 2024-04-25T18:47:42.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff15f400640 10 client.25366 handle_command_reply: tid=0 2024-04-25T18:47:42.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 client.25366 shutdown 2024-04-25T18:47:42.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 msgr2=0x565211836180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:42.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff164000c00 tx=0x7ff1640b9a00 comp rx=0 tx=0).stop 2024-04-25T18:47:42.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 shutdown_connections 2024-04-25T18:47:42.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5652118a7800 0x5652118a9c20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x565211830a00 0x5652118366c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x565211826140 0x565211836180 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 --2- 172.21.15.63:0/647324703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x565211825820 0x565211838df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.049+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 >> 172.21.15.63:0/647324703 conn(0x565211821d30 msgr2=0x565211827a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff15f400640 0 client.25366 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 shutdown_connections 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff1833a4740 1 -- 172.21.15.63:0/647324703 wait complete. 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff1833a4740 20 client.25366 trim_cache size 0 max 16384 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:47:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:47:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 msgr2=0x7ff1600a64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.050+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 0x7ff1600a64d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff170005f40 tx=0x7ff1700096a0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 msgr2=0x7ff17810dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:42.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff17810dfa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff16402a910 tx=0x7ff164003cd0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 shutdown_connections 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff1600a4010 0x7ff1600a64d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff17810e4e0 0x7ff17810e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff178109470 0x7ff17810dfa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 --2- 172.21.15.63:0/4251338667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1781043a0 0x7ff178110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 >> 172.21.15.63:0/4251338667 conn(0x7ff1780ffc40 msgr2=0x7ff178101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 shutdown_connections 2024-04-25T18:47:42.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:42.051+0000 7ff180600640 1 -- 172.21.15.63:0/4251338667 wait complete. 2024-04-25T18:47:42.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:42 smithi114 ceph-mon[32988]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T18:47:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:42 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:42 smithi063 ceph-mon[28006]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T18:47:42.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:42 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:42 smithi070 ceph-mon[32820]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-04-25T18:47:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:42 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:47:43.034 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '55d5471', quiesce took 8.1 seconds. Will release after: 10 2024-04-25T18:47:43.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:43 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:43.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:43 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:43.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:43 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:47:43.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:43 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:43.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:43 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:43.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:43 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:47:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:43 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:47:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:43 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:47:43.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:43 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:47:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:44 smithi114 ceph-mon[32988]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T18:47:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:44 smithi063 ceph-mon[28006]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T18:47:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:44 smithi070 ceph-mon[32820]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-25T18:47:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:46 smithi114 ceph-mon[32988]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-25T18:47:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:46 smithi063 ceph-mon[28006]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-25T18:47:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:46 smithi070 ceph-mon[32820]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-25T18:47:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:48 smithi114 ceph-mon[32988]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:48.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:48 smithi063 ceph-mon[28006]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:48 smithi070 ceph-mon[32820]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:49.778 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:49 smithi063 ceph-mon[28006]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:49.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:49 smithi114 ceph-mon[32988]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:49.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:49 smithi070 ceph-mon[32820]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-25T18:47:51.993 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:51 smithi063 ceph-mon[28006]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-25T18:47:52.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:51 smithi114 ceph-mon[32988]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-25T18:47:52.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:51 smithi070 ceph-mon[32820]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-04-25T18:47:53.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:47:53.571 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:54.071 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:53 smithi063 ceph-mon[28006]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-25T18:47:54.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:53 smithi114 ceph-mon[32988]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-25T18:47:54.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:53 smithi070 ceph-mon[32820]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-04-25T18:47:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 -- 172.21.15.63:0/173326618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed4103830 msgr2=0x7fbed410bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 --2- 172.21.15.63:0/173326618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed4103830 0x7fbed410bd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009860 tx=0x7fbec002a3d0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 -- 172.21.15.63:0/173326618 shutdown_connections 2024-04-25T18:47:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 --2- 172.21.15.63:0/173326618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed4103830 0x7fbed410bd10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 --2- 172.21.15.63:0/173326618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed410c330 0x7fbed410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 --2- 172.21.15.63:0/173326618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4102f10 0x7fbed41032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 -- 172.21.15.63:0/173326618 >> 172.21.15.63:0/173326618 conn(0x7fbed40fc770 msgr2=0x7fbed40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 -- 172.21.15.63:0/173326618 shutdown_connections 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.194+0000 7fbed9400640 1 -- 172.21.15.63:0/173326618 wait complete. 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.195+0000 7fbed9400640 1 Processor -- start 2024-04-25T18:47:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 -- start start 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 0x7fbed41a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed410c330 0x7fbed419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbed4112010 con 0x7fbed4102f10 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbed4111e90 con 0x7fbed410c330 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed9400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbed4112190 con 0x7fbed4103830 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 0x7fbed41a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.196+0000 7fbed3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59168/0 (socket says 172.21.15.63:59168) 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3400640 1 -- 172.21.15.63:0/988682006 learned_addr learned my addr 172.21.15.63:0/988682006 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed8a00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed410c330 0x7fbed419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 -- 172.21.15.63:0/988682006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 msgr2=0x7fbed41a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 -- 172.21.15.63:0/988682006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed410c330 msgr2=0x7fbed419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed410c330 0x7fbed419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 -- 172.21.15.63:0/988682006 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed419ad90 con 0x7fbed4102f10 2024-04-25T18:47:55.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.197+0000 7fbed3e00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 0x7fbed41a0410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009fd0 tx=0x7fbec400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.198+0000 7fbed0c00640 1 -- 172.21.15.63:0/988682006 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4004220 con 0x7fbed4102f10 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.198+0000 7fbed0c00640 1 -- 172.21.15.63:0/988682006 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec40043c0 con 0x7fbed4102f10 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.198+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed419b080 con 0x7fbed4102f10 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.198+0000 7fbed0c00640 1 -- 172.21.15.63:0/988682006 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec40047a0 con 0x7fbed4102f10 2024-04-25T18:47:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.198+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed41a72d0 con 0x7fbed4102f10 2024-04-25T18:47:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.201+0000 7fbedd7c5740 1 Processor -- start 2024-04-25T18:47:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.201+0000 7fbedd7c5740 1 -- start start 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56062493f900 con 0x5606249da2d0 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5606248e5ef0 con 0x5606249d99b0 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56062493b8c0 con 0x5606249e4b90 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34654/0 (socket says 172.21.15.63:34654) 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 -- 172.21.15.63:0/3431212704 learned_addr learned my addr 172.21.15.63:0/3431212704 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed3400640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed0c00640 1 -- 172.21.15.63:0/988682006 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7fbec400c8b0 con 0x7fbed4102f10 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 -- 172.21.15.63:0/3431212704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 msgr2=0x5606249e6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.202+0000 7fbed8a00640 1 -- 172.21.15.63:0/3431212704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 msgr2=0x5606249e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbed3e00640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbed8a00640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbed8a00640 1 -- 172.21.15.63:0/3431212704 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560624652210 con 0x5606249d99b0 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbed3e00640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbed8a00640 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009f60 tx=0x7fbecc009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=181137ddc4911a83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.203+0000 7fbec9200640 1 -- 172.21.15.63:0/3431212704 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01e070 con 0x5606249d99b0 2024-04-25T18:47:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbec9200640 1 -- 172.21.15.63:0/3431212704 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbecc003c30 con 0x5606249d99b0 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbec9200640 1 -- 172.21.15.63:0/3431212704 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00f820 con 0x5606249d99b0 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbed0c00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 0x7fbeb80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbed0c00640 1 -- 172.21.15.63:0/988682006 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fbec40d0a70 con 0x7fbed4102f10 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbed3e00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 0x7fbeb80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3431212704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 msgr2=0x5606249d9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.204+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009f60 tx=0x7fbecc009f90 comp rx=0 tx=0).stop 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3431212704 shutdown_connections 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3431212704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d9d90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3431212704 >> 172.21.15.63:0/3431212704 conn(0x5606249d5ec0 msgr2=0x5606249d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3431212704 shutdown_connections 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3431212704 wait complete. 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbed3e00640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 0x7fbeb80a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbec40ce030 tx=0x7fbec4005230 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 -- start start 2024-04-25T18:47:55.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.205+0000 7fbedd7c5740 1 Processor -- start 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249e1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5606249e77c0 con 0x5606249da2d0 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5606248f9310 con 0x5606249d99b0 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5606249e7940 con 0x5606249e4b90 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbedd7c5740 10 client.0 _ll_register_callbacks cb 0x5606248c7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249e1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbed3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249e1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34664/0 (socket says 172.21.15.63:34664) 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbed8a00640 1 -- 172.21.15.63:0/985255213 learned_addr learned my addr 172.21.15.63:0/985255213 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.206+0000 7fbed8a00640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 -- 172.21.15.63:0/985255213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 msgr2=0x5606249e25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 -- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 msgr2=0x5606249e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249e1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 -- 172.21.15.63:0/985255213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062471e620 con 0x5606249da2d0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbed3400640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e00 tx=0x7fbec0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef69214980700375 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbebd600640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0030750 con 0x5606249da2d0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fbec00055f0 con 0x5606249da2d0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.207+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbecc003c30 con 0x5606249da2d0 2024-04-25T18:47:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbec0004050 con 0x5606249da2d0 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbedd7c5740 10 client.15204 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5606249f39b0 con 0x5606249da2d0 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbecc003c30 con 0x5606249da2d0 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fbec00035f0 con 0x5606249da2d0 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbedd7c5740 10 client.15204 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbedd7c5740 10 client.15204 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.208+0000 7fbedd7c5740 20 client.15204 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5606249f4e20 0x5606249f7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbedd7c5740 4 client.15204 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606249538b0 con 0x5606249f4e20 2024-04-25T18:47:55.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbed8a00640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5606249f4e20 0x5606249f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbed8a00640 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5606249f4e20 0x5606249f7210 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5043d76beb321a server_cookie=561e833601f7ee8b in_seq=0 out_seq=0 2024-04-25T18:47:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.209+0000 7fbebd600640 10 client.15204 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.210+0000 7fbebd600640 1 -- 172.21.15.63:0/985255213 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606249538b0 con 0x5606249f4e20 2024-04-25T18:47:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.210+0000 7fbebd600640 10 client.15204 handle_command_reply: tid=0 2024-04-25T18:47:55.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.210+0000 7fbedd7c5740 1 client.15204 shutdown 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 msgr2=0x5606249e2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e00 tx=0x7fbec0005270 comp rx=0 tx=0).stop 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 shutdown_connections 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5606249f4e20 0x5606249f7210 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249e25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249e2080 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbebd600640 0 client.15204 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/985255213 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249e1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 >> 172.21.15.63:0/985255213 conn(0x5606249d5ec0 msgr2=0x5606249dbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 shutdown_connections 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 1 -- 172.21.15.63:0/985255213 wait complete. 2024-04-25T18:47:55.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.211+0000 7fbedd7c5740 20 client.15204 trim_cache size 0 max 16384 2024-04-25T18:47:55.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.227+0000 7fbedd7c5740 1 Processor -- start 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.227+0000 7fbedd7c5740 1 -- start start 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41b2530 con 0x5606249da2d0 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41afe00 con 0x5606249d99b0 2024-04-25T18:47:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41adc50 con 0x5606249e4b90 2024-04-25T18:47:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41360/0 (socket says 172.21.15.63:41360) 2024-04-25T18:47:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 -- 172.21.15.63:0/3821114146 learned_addr learned my addr 172.21.15.63:0/3821114146 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed8a00640 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed8a00640 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 -- 172.21.15.63:0/3821114146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 msgr2=0x5606249fcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249fcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.228+0000 7fbed3400640 1 -- 172.21.15.63:0/3821114146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 msgr2=0x5606249d57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbed3400640 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbed3400640 1 -- 172.21.15.63:0/3821114146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606249f7870 con 0x5606249da2d0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbed3400640 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbec0004020 tx=0x7fbec0005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c51d20f6e40a75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbebfe00640 1 -- 172.21.15.63:0/3821114146 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0052030 con 0x5606249da2d0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbebfe00640 1 -- 172.21.15.63:0/3821114146 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec0004e70 con 0x5606249da2d0 2024-04-25T18:47:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.229+0000 7fbebfe00640 1 -- 172.21.15.63:0/3821114146 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0004b70 con 0x5606249da2d0 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3821114146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 msgr2=0x5606249dc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbec0004020 tx=0x7fbec0005af0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3821114146 shutdown_connections 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249e4b90 0x5606249fcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249da2d0 0x5606249dc370 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/3821114146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249d57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3821114146 >> 172.21.15.63:0/3821114146 conn(0x5606249d5ec0 msgr2=0x5606249e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3821114146 shutdown_connections 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.230+0000 7fbedd7c5740 1 -- 172.21.15.63:0/3821114146 wait complete. 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 -- start start 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 Processor -- start 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249e4b90 0x5606249eb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41ae2c0 con 0x5606249e4b90 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41af730 con 0x5606249d99b0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbed41ae990 con 0x5606249da2d0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.231+0000 7fbedd7c5740 10 client.0 _ll_register_callbacks cb 0x7fbec0030dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249e4b90 0x5606249eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59210/0 (socket says 172.21.15.63:59210) 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed8a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249ea7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34684/0 (socket says 172.21.15.63:34684) 2024-04-25T18:47:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 -- 172.21.15.63:0/2454771341 learned_addr learned my addr 172.21.15.63:0/2454771341 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 -- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 msgr2=0x5606249ea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249ea7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 -- 172.21.15.63:0/2454771341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249e4b90 msgr2=0x5606249eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249e4b90 0x5606249eb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.232+0000 7fbed3400640 1 -- 172.21.15.63:0/2454771341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560624a5a5d0 con 0x5606249da2d0 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbed3400640 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e00 tx=0x7fbec0003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe8e1ba3cd72179 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbebd600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00096b0 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec0004e70 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fbec004a020 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbec0066050 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbedd7c5740 10 client.25384 fetch_fsmap learned FSMap version 24 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560624a5b880 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.233+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec0004e70 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fbec005f060 con 0x5606249da2d0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 10 client.25384 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 10 client.25384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 20 client.25384 populate_metadata read hostname 'smithi063' 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560624a5beb0 0x560624a5e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 4 client.25384 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56062494d320 con 0x560624a5beb0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.234+0000 7fbed8a00640 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560624a5beb0 0x560624a5e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.235+0000 7fbebd600640 10 client.25384 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.235+0000 7fbed8a00640 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560624a5beb0 0x560624a5e2d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d26c8620dc0510df server_cookie=f6e55343b7e9ad0c in_seq=0 out_seq=0 2024-04-25T18:47:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbebd600640 1 -- 172.21.15.63:0/2454771341 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56062494d320 con 0x560624a5beb0 2024-04-25T18:47:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbebd600640 10 client.25384 handle_command_reply: tid=0 2024-04-25T18:47:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 client.25384 shutdown 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 msgr2=0x5606249ead10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbec0005e00 tx=0x7fbec0003ec0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 shutdown_connections 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560624a5beb0 0x560624a5e2d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606249e4b90 0x5606249eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbebd600640 0 client.25384 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5606249da2d0 0x5606249ead10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 --2- 172.21.15.63:0/2454771341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5606249d99b0 0x5606249ea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 >> 172.21.15.63:0/2454771341 conn(0x5606249d5ec0 msgr2=0x5606249e1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 shutdown_connections 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 1 -- 172.21.15.63:0/2454771341 wait complete. 2024-04-25T18:47:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.241+0000 7fbedd7c5740 20 client.25384 trim_cache size 0 max 16384 2024-04-25T18:47:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:47:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:47:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:47:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:47:55.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 msgr2=0x7fbeb80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 0x7fbeb80a62f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbec40ce030 tx=0x7fbec4005230 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 msgr2=0x7fbed41a0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 0x7fbed41a0410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009fd0 tx=0x7fbec400eb60 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 shutdown_connections 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeb80a3e30 0x7fbeb80a62f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbed410c330 0x7fbed419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbed4103830 0x7fbed41a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 --2- 172.21.15.63:0/988682006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4102f10 0x7fbed41a0410 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.242+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 >> 172.21.15.63:0/988682006 conn(0x7fbed40fc770 msgr2=0x7fbed4101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.243+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 shutdown_connections 2024-04-25T18:47:55.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:47:55.243+0000 7fbed9400640 1 -- 172.21.15.63:0/988682006 wait complete. 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 -- 172.21.15.63:0/463483099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd806b980 msgr2=0x7f6fd810f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 --2- 172.21.15.63:0/463483099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd806b980 0x7f6fd810f8f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6fd400b0a0 tx=0x7f6fd400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 -- 172.21.15.63:0/463483099 shutdown_connections 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 --2- 172.21.15.63:0/463483099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd806b980 0x7f6fd810f8f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 --2- 172.21.15.63:0/463483099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd8108a20 0x7f6fd8108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 --2- 172.21.15.63:0/463483099 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806afe0 0x7f6fd806b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 -- 172.21.15.63:0/463483099 >> 172.21.15.63:0/463483099 conn(0x7f6fd80788b0 msgr2=0x7f6fd8078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:56.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 -- 172.21.15.63:0/463483099 shutdown_connections 2024-04-25T18:47:56.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.234+0000 7f6fe0800700 1 -- 172.21.15.63:0/463483099 wait complete. 2024-04-25T18:47:56.237 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:56 smithi063 ceph-mon[28006]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-25T18:47:56.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.236+0000 7f6fe0800700 1 Processor -- start 2024-04-25T18:47:56.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 -- start start 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 0x7f6fd819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 0x7f6fd819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112c10 con 0x7f6fd806afe0 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112a90 con 0x7f6fd8108a20 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.237+0000 7f6fe0800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112d90 con 0x7f6fd806b980 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:59228/0 (socket says 172.21.15.63:59228) 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 -- 172.21.15.63:0/3283455885 learned_addr learned my addr 172.21.15.63:0/3283455885 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:47:56.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdde00700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 0x7f6fd819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:56.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fde800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 0x7f6fd819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:56.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 -- 172.21.15.63:0/3283455885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 msgr2=0x7f6fd819fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:56.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 0x7f6fd819fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 -- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 msgr2=0x7f6fd819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 0x7f6fd819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdd400700 1 -- 172.21.15.63:0/3283455885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd81a02f0 con 0x7f6fd806b980 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fde800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 0x7f6fd819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.238+0000 7f6fdde00700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 0x7f6fd819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fdd400700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800e9a0 tx=0x7f6fc800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc800cbd0 con 0x7f6fd806b980 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fc800cd70 con 0x7f6fd806b980 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd81a05e0 con 0x7f6fd806b980 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8018660 con 0x7f6fd806b980 2024-04-25T18:47:56.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.239+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd81a7e60 con 0x7f6fd806b980 2024-04-25T18:47:56.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.240+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd8069730 con 0x7f6fd806b980 2024-04-25T18:47:56.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.242+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f6fc8010970 con 0x7f6fd806b980 2024-04-25T18:47:56.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.244+0000 7f6fd2a00700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 0x7f6fc00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:47:56.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.244+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f6fc80d0e40 con 0x7f6fd806b980 2024-04-25T18:47:56.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.244+0000 7f6fdde00700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 0x7f6fc00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:47:56.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.245+0000 7f6fdde00700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 0x7f6fc00a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009c90 tx=0x7f6fcc009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:47:56.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.248+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6fc809d220 con 0x7f6fd806b980 2024-04-25T18:47:56.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:56 smithi114 ceph-mon[32988]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-25T18:47:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:56 smithi070 ceph-mon[32820]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-25T18:47:56.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.529+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fd802df80 con 0x7f6fd806b980 2024-04-25T18:47:56.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.531+0000 7f6fd2a00700 1 -- 172.21.15.63:0/3283455885 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11868 (secure 0 0 0) 0x7f6fc809cb20 con 0x7f6fd806b980 2024-04-25T18:47:56.532 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:47:56.533 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:46:32.730913+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:47:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 msgr2=0x7f6fc00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 0x7f6fc00a6390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009c90 tx=0x7f6fcc009260 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 msgr2=0x7f6fd819b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800e9a0 tx=0x7f6fc800ed60 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 shutdown_connections 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6fc00a3ed0 0x7f6fc00a6390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fd8108a20 0x7f6fd819fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6fd806afe0 0x7f6fd819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 --2- 172.21.15.63:0/3283455885 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6fd806b980 0x7f6fd819b820 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 >> 172.21.15.63:0/3283455885 conn(0x7f6fd80788b0 msgr2=0x7f6fd810de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 shutdown_connections 2024-04-25T18:47:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:47:56.535+0000 7f6fe0800700 1 -- 172.21.15.63:0/3283455885 wait complete. 2024-04-25T18:47:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 24 2024-04-25T18:47:57.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:57 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3283455885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:57.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:57 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3283455885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:57.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:57 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3283455885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:47:57.580 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id 55d5471 --release --await' 2024-04-25T18:47:57.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id 55d5471 --release --await 2024-04-25T18:47:57.746 INFO:tasks.workunit.client.0.smithi063.stderr:Updating files: 58% (7360/12492) Updating files: 59% (7371/12492) Updating files: 60% (7496/12492) Updating files: 61% (7621/12492) Updating files: 62% (7746/12492) Updating files: 63% (7870/12492) Updating files: 64% (7995/12492) Updating files: 65% (8120/12492) Updating files: 66% (8245/12492) Updating files: 67% (8370/12492) Updating files: 68% (8495/12492) Updating files: 69% (8620/12492) Updating files: 70% (8745/12492) Updating files: 71% (8870/12492) Updating files: 72% (8995/12492) Updating files: 73% (9120/12492) Updating files: 74% (9245/12492) Updating files: 75% (9369/12492) Updating files: 76% (9494/12492) Updating files: 77% (9619/12492) Updating files: 78% (9744/12492) Updating files: 79% (9869/12492) Updating files: 80% (9994/12492) Updating files: 81% (10119/12492) Updating files: 82% (10244/12492) Updating files: 83% (10369/12492) Updating files: 84% (10494/12492) Updating files: 85% (10619/12492) Updating files: 86% (10744/12492) Updating files: 87% (10869/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 94% (11834/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T18:47:58.013 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:47:58.312 INFO:tasks.workunit.client.0.smithi063.stderr:Note: switching to '7797745e6d8b3ac036504dc61491d368615024da'. 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: git switch -c 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr:Or undo this operation with: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.313 INFO:tasks.workunit.client.0.smithi063.stderr: git switch - 2024-04-25T18:47:58.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.314 INFO:tasks.workunit.client.0.smithi063.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T18:47:58.314 INFO:tasks.workunit.client.0.smithi063.stderr: 2024-04-25T18:47:58.314 INFO:tasks.workunit.client.0.smithi063.stderr:HEAD is now at 7797745e6d8 qa/suites/fs: add quiescer to the fs suite 2024-04-25T18:47:58.317 DEBUG:teuthology.orchestra.run.smithi063:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T18:47:58.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:47:58 smithi114 ceph-mon[32988]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:47:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:47:58 smithi063 ceph-mon[28006]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:47:58.425 INFO:tasks.workunit.client.0.smithi063.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T18:47:58.432 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T18:47:58.432 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T18:47:58.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:47:58 smithi070 ceph-mon[32820]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-25T18:48:00.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:00 smithi063 ceph-mon[28006]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T18:48:00.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:00 smithi114 ceph-mon[32988]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T18:48:00.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:00 smithi070 ceph-mon[32820]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-25T18:48:00.574 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T18:48:00.773 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T18:48:00.931 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T18:48:00.938 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T18:48:00.938 INFO:tasks.workunit.client.0.smithi063.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T18:48:01.097 INFO:tasks.workunit.client.0.smithi063.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T18:48:01.105 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T18:48:01.105 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 -- 172.21.15.63:0/2809917957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 msgr2=0x7f613c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 --2- 172.21.15.63:0/2809917957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c102f10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f6128009ca0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 -- 172.21.15.63:0/2809917957 shutdown_connections 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 --2- 172.21.15.63:0/2809917957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c103450 0x7f613c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 --2- 172.21.15.63:0/2809917957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c108a80 0x7f613c108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 --2- 172.21.15.63:0/2809917957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c102f10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.986+0000 7f6141c00700 1 -- 172.21.15.63:0/2809917957 >> 172.21.15.63:0/2809917957 conn(0x7f613c079bf0 msgr2=0x7f613c07a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.987+0000 7f6141c00700 1 -- 172.21.15.63:0/2809917957 shutdown_connections 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.987+0000 7f6141c00700 1 -- 172.21.15.63:0/2809917957 wait complete. 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 Processor -- start 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 -- start start 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 0x7f613c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c108a80 0x7f613c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f613c112c80 con 0x7f613c102a90 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f613c112b00 con 0x7f613c103450 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.988+0000 7f6141c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f613c112e00 con 0x7f613c108a80 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35282/0 (socket says 172.21.15.63:35282) 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 -- 172.21.15.63:0/4190507826 learned_addr learned my addr 172.21.15.63:0/4190507826 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613a000700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 0x7f613c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613b400700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c108a80 0x7f613c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 -- 172.21.15.63:0/4190507826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c108a80 msgr2=0x7f613c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c108a80 0x7f613c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 -- 172.21.15.63:0/4190507826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 msgr2=0x7f613c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 0x7f613c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613aa00700 1 -- 172.21.15.63:0/4190507826 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613c1a7670 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.989+0000 7f613a000700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 0x7f613c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f613aa00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9a0 tx=0x7f612c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f6133400700 1 -- 172.21.15.63:0/4190507826 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c00ca10 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f6133400700 1 -- 172.21.15.63:0/4190507826 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612c00cbb0 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f6133400700 1 -- 172.21.15.63:0/4190507826 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c018680 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f613c1a7960 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.990+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f613c1a7e50 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.994+0000 7f6133400700 1 -- 172.21.15.63:0/4190507826 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 25) ==== 138879+0+0 (secure 0 0 0) 0x7f612c010430 con 0x7f613c102a90 2024-04-25T18:48:01.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.996+0000 7f614c5aeb80 1 Processor -- start 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.996+0000 7f614c5aeb80 1 -- start start 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f6133400700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 0x7f61200a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca60eef90 con 0x55cca619c240 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cca6095830 con 0x55cca619b920 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cca60eaf50 con 0x55cca61a70f0 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f6133400700 1 -- 172.21.15.63:0/4190507826 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f612c0d0bc0 con 0x7f613c102a90 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:48448/0 (socket says 172.21.15.63:48448) 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613aa00700 1 -- 172.21.15.63:0/2301498375 learned_addr learned my addr 172.21.15.63:0/2301498375 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:01.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613b400700 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.997+0000 7f613b400700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 0x7f61200a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 -- 172.21.15.63:0/2301498375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 msgr2=0x55cca61a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61a94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 -- 172.21.15.63:0/2301498375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 msgr2=0x55cca61a6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61a6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 -- 172.21.15.63:0/2301498375 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca608b780 con 0x55cca619b920 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613a000700 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f613aa00700 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f612c09c830 tx=0x7f612c09cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5d8b256f41def2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f6127e00700 1 -- 172.21.15.63:0/2301498375 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c0a5510 con 0x55cca619b920 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f6127e00700 1 -- 172.21.15.63:0/2301498375 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612c09a940 con 0x55cca619b920 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.998+0000 7f6127e00700 1 -- 172.21.15.63:0/2301498375 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c09cd60 con 0x55cca619b920 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f613b400700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 0x7f61200a6410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6134009e10 tx=0x7f613400b040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 -- 172.21.15.63:0/2301498375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 msgr2=0x55cca619bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f612c09c830 tx=0x7f612c09cfa0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 -- 172.21.15.63:0/2301498375 shutdown_connections 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61a94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619bd00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/2301498375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:00.999+0000 7f614c5aeb80 1 -- 172.21.15.63:0/2301498375 >> 172.21.15.63:0/2301498375 conn(0x55cca60ca6a0 msgr2=0x55cca6197dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.000+0000 7f614c5aeb80 1 -- 172.21.15.63:0/2301498375 shutdown_connections 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.000+0000 7f614c5aeb80 1 -- 172.21.15.63:0/2301498375 wait complete. 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.000+0000 7f614c5aeb80 1 -- start start 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.000+0000 7f614c5aeb80 1 Processor -- start 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619c240 0x55cca61a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca61a9c10 con 0x55cca61a70f0 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cca60a8960 con 0x55cca619c240 2024-04-25T18:48:01.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cca61a9d90 con 0x55cca619b920 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f614c5aeb80 10 client.0 _ll_register_callbacks cb 0x55cca6087820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f613aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f613aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55290/0 (socket says 172.21.15.63:55290) 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.001+0000 7f613aa00700 1 -- 172.21.15.63:0/4210664062 learned_addr learned my addr 172.21.15.63:0/4210664062 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613a000700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619c240 0x55cca61a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613b400700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 -- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619c240 msgr2=0x55cca61a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619c240 0x55cca61a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 -- 172.21.15.63:0/4210664062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 msgr2=0x55cca61a40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 -- 172.21.15.63:0/4210664062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca603c9d0 con 0x55cca619b920 2024-04-25T18:48:01.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f613aa00700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f612c09c370 tx=0x7f612c010c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37e56ca28f1cd68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f6124c00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c0a5510 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f612c09dbd0 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f612c09a940 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f612c09ddd0 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f614c5aeb80 10 client.25402 fetch_fsmap learned FSMap version 24 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.002+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cca6085bf0 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612c09a940 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f612c010720 con 0x55cca619b920 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 10 client.25402 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 10 client.25402 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 20 client.25402 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca61b67c0 0x55cca61b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 4 client.25402 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:01.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cca60fc780 con 0x55cca61b67c0 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.003+0000 7f613a000700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca61b67c0 0x55cca61b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.004+0000 7f613a000700 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca61b67c0 0x55cca61b8bb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a660c2619b7f5d5 server_cookie=aabe036296ab1e4d in_seq=0 out_seq=0 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.004+0000 7f6124c00700 10 client.25402 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f6124c00700 1 -- 172.21.15.63:0/4210664062 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cca60fc780 con 0x55cca61b67c0 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f6124c00700 10 client.25402 handle_command_reply: tid=0 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 client.25402 shutdown 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 msgr2=0x55cca61a6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f612c09c370 tx=0x7f612c010c10 comp rx=0 tx=0).stop 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 shutdown_connections 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca61b67c0 0x55cca61b8bb0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619b920 0x55cca61a6860 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/4210664062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619c240 0x55cca61a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f6124c00700 0 client.25402 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.005+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 >> 172.21.15.63:0/4210664062 conn(0x55cca60ca6a0 msgr2=0x55cca61979c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.006+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 shutdown_connections 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.006+0000 7f614c5aeb80 1 -- 172.21.15.63:0/4210664062 wait complete. 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.006+0000 7f614c5aeb80 20 client.25402 trim_cache size 0 max 16384 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 Processor -- start 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 -- start start 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1b1fc0 con 0x55cca61a70f0 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1af850 con 0x55cca619b920 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.028+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1ad6a0 con 0x55cca619c240 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55300/0 (socket says 172.21.15.63:55300) 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 -- 172.21.15.63:0/1383157399 learned_addr learned my addr 172.21.15.63:0/1383157399 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613aa00700 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:48454/0 (socket says 172.21.15.63:48454) 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613b400700 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 -- 172.21.15.63:0/1383157399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 msgr2=0x55cca619f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 -- 172.21.15.63:0/1383157399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 msgr2=0x55cca61a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.029+0000 7f613a000700 1 -- 172.21.15.63:0/1383157399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca619c620 con 0x55cca619c240 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.030+0000 7f613a000700 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f612801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc9951ce393a3a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.030+0000 7f6127400700 1 -- 172.21.15.63:0/1383157399 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128005690 con 0x55cca619c240 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.030+0000 7f6127400700 1 -- 172.21.15.63:0/1383157399 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612802d070 con 0x55cca619c240 2024-04-25T18:48:01.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.030+0000 7f6127400700 1 -- 172.21.15.63:0/1383157399 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61280039b0 con 0x55cca619c240 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- 172.21.15.63:0/1383157399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 msgr2=0x55cca6197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6128009990 tx=0x7f612801a040 comp rx=0 tx=0).stop 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- 172.21.15.63:0/1383157399 shutdown_connections 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca61a70f0 0x55cca61a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca619f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/1383157399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca619c240 0x55cca6197f10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- 172.21.15.63:0/1383157399 >> 172.21.15.63:0/1383157399 conn(0x55cca60ca6a0 msgr2=0x55cca61a5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- 172.21.15.63:0/1383157399 shutdown_connections 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- 172.21.15.63:0/1383157399 wait complete. 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.031+0000 7f614c5aeb80 1 -- start start 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 Processor -- start 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca61ba6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61bb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1add10 con 0x55cca619c240 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1af180 con 0x55cca619b920 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f613c1ae3e0 con 0x55cca61a70f0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.032+0000 7f614c5aeb80 10 client.0 _ll_register_callbacks cb 0x55cca6087820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35314/0 (socket says 172.21.15.63:35314) 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613b400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61bb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 -- 172.21.15.63:0/3975041986 learned_addr learned my addr 172.21.15.63:0/3975041986 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613aa00700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca61ba6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 -- 172.21.15.63:0/3975041986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 msgr2=0x55cca61bb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61bb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 -- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 msgr2=0x55cca61ba6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca61ba6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 -- 172.21.15.63:0/3975041986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca621cae0 con 0x55cca619c240 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f613a000700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6128009ce0 tx=0x7f612800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2b60298cc06c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:01.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.033+0000 7f6124c00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128028360 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f6128028500 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f612802d070 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6128041050 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f614c5aeb80 10 client.15231 fetch_fsmap learned FSMap version 24 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cca621de20 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.034+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612802d070 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f612800f4f0 con 0x55cca619c240 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 10 client.15231 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 10 client.15231 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 20 client.15231 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca621e500 0x55cca6220920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 4 client.15231 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "55d5471", "release": true, "await": true}] 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "55d5471", "release": true, "await": true}) -- 0x55cca60f6820 con 0x55cca621e500 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.035+0000 7f613aa00700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca621e500 0x55cca6220920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:01.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.036+0000 7f613aa00700 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca621e500 0x55cca6220920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=914a0583e03a84bb server_cookie=84a2c4a5a1afdbef in_seq=0 out_seq=0 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.036+0000 7f6124c00700 10 client.15231 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f6124c00700 1 -- 172.21.15.63:0/3975041986 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+587 (crc 0 0 0) 0x55cca60f6820 con 0x55cca621e500 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f6124c00700 10 client.15231 handle_command_reply: tid=0 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 client.15231 shutdown 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 msgr2=0x55cca61bac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6128009ce0 tx=0x7f612800b410 comp rx=0 tx=0).stop 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 shutdown_connections 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cca621e500 0x55cca6220920 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cca61a70f0 0x55cca61bb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cca619b920 0x55cca61ba6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 --2- 172.21.15.63:0/3975041986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca619c240 0x55cca61bac20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f6124c00700 0 client.15231 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 >> 172.21.15.63:0/3975041986 conn(0x55cca60ca6a0 msgr2=0x55cca619d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 shutdown_connections 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 1 -- 172.21.15.63:0/3975041986 wait complete. 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 24, 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 4, 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "55d5471": { 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 4, 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:48:01.135 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 70.0, 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:01.136 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.038+0000 7f614c5aeb80 20 client.15231 trim_cache size 0 max 16384 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 msgr2=0x7f61200a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 0x7f61200a6410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6134009e10 tx=0x7f613400b040 comp rx=0 tx=0).stop 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 msgr2=0x7f613c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9a0 tx=0x7f612c00ecb0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 shutdown_connections 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f61200a3f50 0x7f61200a6410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f613c108a80 0x7f613c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f613c102a90 0x7f613c074d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 --2- 172.21.15.63:0/4190507826 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f613c103450 0x7f613c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:01.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 >> 172.21.15.63:0/4190507826 conn(0x7f613c079bf0 msgr2=0x7f613c101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:01.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.039+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 shutdown_connections 2024-04-25T18:48:01.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:01.040+0000 7f6141c00700 1 -- 172.21.15.63:0/4190507826 wait complete. 2024-04-25T18:48:01.161 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-25T18:48:01.163 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-25T18:48:01.163 DEBUG:teuthology.orchestra.run.smithi063:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7797745e6d8b3ac036504dc61491d368615024da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-25T18:48:01.277 INFO:tasks.workunit.client.0.smithi063.stderr:+ dbench 1 2024-04-25T18:48:01.305 INFO:tasks.workunit.client.0.smithi063.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-25T18:48:01.305 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-04-25T18:48:01.306 INFO:tasks.workunit.client.0.smithi063.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-25T18:48:01.329 INFO:tasks.workunit.client.0.smithi063.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-25T18:48:01.429 INFO:tasks.workunit.client.0.smithi063.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-25T18:48:01.429 INFO:tasks.workunit.client.0.smithi063.stdout:releasing clients 2024-04-25T18:48:02.164 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '55d5471', seconds elapsed: 27.3 2024-04-25T18:48:02.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:02 smithi114 ceph-mon[32988]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:48:02.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:02 smithi063 ceph-mon[28006]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:48:02.429 INFO:tasks.workunit.client.0.smithi063.stdout: 1 1758 26.28 MB/sec warmup 1 sec latency 38.426 ms 2024-04-25T18:48:02.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:02 smithi070 ceph-mon[32820]: pgmap v47: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-04-25T18:48:03.429 INFO:tasks.workunit.client.0.smithi063.stdout: 1 3974 20.03 MB/sec warmup 2 sec latency 12.218 ms 2024-04-25T18:48:04.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:04 smithi114 ceph-mon[32988]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:48:04.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:04 smithi063 ceph-mon[28006]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:48:04.429 INFO:tasks.workunit.client.0.smithi063.stdout: 1 6012 15.74 MB/sec warmup 3 sec latency 11.924 ms 2024-04-25T18:48:04.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:04 smithi070 ceph-mon[32820]: pgmap v48: 97 pgs: 97 active+clean; 1.4 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-25T18:48:05.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 7824 14.86 MB/sec warmup 4 sec latency 13.717 ms 2024-04-25T18:48:06.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:06 smithi114 ceph-mon[32988]: pgmap v49: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 295 KiB/s wr, 53 op/s 2024-04-25T18:48:06.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:06 smithi114 ceph-mon[32988]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T18:48:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:06 smithi063 ceph-mon[28006]: pgmap v49: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 295 KiB/s wr, 53 op/s 2024-04-25T18:48:06.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:06 smithi063 ceph-mon[28006]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T18:48:06.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 10081 14.35 MB/sec warmup 5 sec latency 19.566 ms 2024-04-25T18:48:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:06 smithi070 ceph-mon[32820]: pgmap v49: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 295 KiB/s wr, 53 op/s 2024-04-25T18:48:06.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:06 smithi070 ceph-mon[32820]: mgrmap e26: x(active, since 92s), standbys: y, z 2024-04-25T18:48:07.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 11627 13.19 MB/sec warmup 6 sec latency 19.057 ms 2024-04-25T18:48:08.316 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:08 smithi114 ceph-mon[32988]: pgmap v50: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 295 KiB/s wr, 50 op/s 2024-04-25T18:48:08.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:08 smithi063 ceph-mon[28006]: pgmap v50: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 295 KiB/s wr, 50 op/s 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.397+0000 7fee14600640 1 -- 172.21.15.63:0/3712823528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c1043b0 msgr2=0x7fee0c104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.397+0000 7fee14600640 1 --2- 172.21.15.63:0/3712823528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c1043b0 0x7fee0c104790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee0002a3f0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 -- 172.21.15.63:0/3712823528 shutdown_connections 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 --2- 172.21.15.63:0/3712823528 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c104d60 0x7fee0c108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 --2- 172.21.15.63:0/3712823528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c109480 0x7fee0c111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 --2- 172.21.15.63:0/3712823528 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c1043b0 0x7fee0c104790 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 -- 172.21.15.63:0/3712823528 >> 172.21.15.63:0/3712823528 conn(0x7fee0c0ffc50 msgr2=0x7fee0c102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 -- 172.21.15.63:0/3712823528 shutdown_connections 2024-04-25T18:48:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.398+0000 7fee14600640 1 -- 172.21.15.63:0/3712823528 wait complete. 2024-04-25T18:48:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.399+0000 7fee14600640 1 Processor -- start 2024-04-25T18:48:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.399+0000 7fee14600640 1 -- start start 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.399+0000 7fee14600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee14600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee14600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c19a710 0x7fee0c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee14600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c114430 con 0x7fee0c1043b0 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee14600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1142b0 con 0x7fee0c109480 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee14600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1145b0 con 0x7fee0c19a710 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39964/0 (socket says 172.21.15.63:39964) 2024-04-25T18:48:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51130/0 (socket says 172.21.15.63:51130) 2024-04-25T18:48:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 -- 172.21.15.63:0/3948154523 learned_addr learned my addr 172.21.15.63:0/3948154523 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee12600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c19a710 0x7fee0c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 -- 172.21.15.63:0/3948154523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c19a710 msgr2=0x7fee0c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c19a710 0x7fee0c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 -- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 msgr2=0x7fee0c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11200640 1 -- 172.21.15.63:0/3948154523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c19b310 con 0x7fee0c109480 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.400+0000 7fee11c00640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee11200640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fedf800e9f0 tx=0x7fedf800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee0a000640 1 -- 172.21.15.63:0/3948154523 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf800ccc0 con 0x7fee0c109480 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee0a000640 1 -- 172.21.15.63:0/3948154523 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedf80054d0 con 0x7fee0c109480 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c1a73f0 con 0x7fee0c109480 2024-04-25T18:48:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee0a000640 1 -- 172.21.15.63:0/3948154523 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf8010430 con 0x7fee0c109480 2024-04-25T18:48:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.401+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c1a77a0 con 0x7fee0c109480 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 Processor -- start 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee0a000640 1 -- 172.21.15.63:0/3948154523 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fedf80046c0 con 0x7fee0c109480 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 -- start start 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e3ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e4bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f92da4900 con 0x561f92e49a00 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f92d4aef0 con 0x561f92e3e820 2024-04-25T18:48:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.405+0000 7fee1742b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f92da08c0 con 0x561f92e3f140 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee12600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e3ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:60470/0 (socket says 172.21.15.63:60470) 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 -- 172.21.15.63:0/3695261396 learned_addr learned my addr 172.21.15.63:0/3695261396 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11200640 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e4bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 -- 172.21.15.63:0/3695261396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 msgr2=0x561f92e3ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e3ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 -- 172.21.15.63:0/3695261396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 msgr2=0x561f92e4bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e4bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 -- 172.21.15.63:0/3695261396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f92ab7210 con 0x561f92e3f140 2024-04-25T18:48:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.406+0000 7fee11c00640 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee00009230 tx=0x7fee00005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cc1b97831e907f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fee0a000640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 0x7fedf40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fedfea00640 1 -- 172.21.15.63:0/3695261396 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00035040 con 0x561f92e3f140 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fedfea00640 1 -- 172.21.15.63:0/3695261396 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee00004c20 con 0x561f92e3f140 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fedfea00640 1 -- 172.21.15.63:0/3695261396 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee000039f0 con 0x561f92e3f140 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fee0a000640 1 -- 172.21.15.63:0/3948154523 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fedf80d0ba0 con 0x7fee0c109480 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.407+0000 7fee12600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 0x7fedf40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 -- 172.21.15.63:0/3695261396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 msgr2=0x561f92e494c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee12600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 0x7fedf40a6280 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04009600 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fee00009230 tx=0x7fee00005c00 comp rx=0 tx=0).stop 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 -- 172.21.15.63:0/3695261396 shutdown_connections 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e4bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e494c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 --2- 172.21.15.63:0/3695261396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e3ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 -- 172.21.15.63:0/3695261396 >> 172.21.15.63:0/3695261396 conn(0x561f92e3ad30 msgr2=0x561f92e3b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 -- 172.21.15.63:0/3695261396 shutdown_connections 2024-04-25T18:48:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.408+0000 7fee1742b740 1 -- 172.21.15.63:0/3695261396 wait complete. 2024-04-25T18:48:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.409+0000 7fee1742b740 1 -- start start 2024-04-25T18:48:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.409+0000 7fee1742b740 1 Processor -- start 2024-04-25T18:48:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e46ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e47430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f92e4c5d0 con 0x561f92e49a00 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f92d5e310 con 0x561f92e3e820 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f92e4c750 con 0x561f92e3f140 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee1742b740 10 client.0 _ll_register_callbacks cb 0x561f92d2ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39982/0 (socket says 172.21.15.63:39982) 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 -- 172.21.15.63:0/3023205424 learned_addr learned my addr 172.21.15.63:0/3023205424 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee12600640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e47430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11200640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e46ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 -- 172.21.15.63:0/3023205424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 msgr2=0x561f92e46ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e46ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 -- 172.21.15.63:0/3023205424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 msgr2=0x561f92e47430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e47430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.410+0000 7fee11c00640 1 -- 172.21.15.63:0/3023205424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f92b83620 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fee11c00640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee00006010 tx=0x7fee00005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=124aa43a7b9dbcd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fedf3400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00053060 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fee00004980 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee00004c20 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee00035050 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fee1742b740 10 client.25397 fetch_fsmap learned FSMap version 24 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.411+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561f92e58850 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee00004c20 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fee000314a0 con 0x561f92e3e820 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 10 client.25397 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 10 client.25397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 20 client.25397 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92e59ba0 0x561f92e5bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 4 client.25397 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.412+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f92db88b0 con 0x561f92e59ba0 2024-04-25T18:48:08.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.413+0000 7fee11200640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92e59ba0 0x561f92e5bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.413+0000 7fee11200640 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92e59ba0 0x561f92e5bf90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed30c45b84975a62 server_cookie=2321a6905ff67cf5 in_seq=0 out_seq=0 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.413+0000 7fedf3400640 10 client.25397 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.414+0000 7fedf3400640 1 -- 172.21.15.63:0/3023205424 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f92db88b0 con 0x561f92e59ba0 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.414+0000 7fedf3400640 10 client.25397 handle_command_reply: tid=0 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 client.25397 shutdown 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 msgr2=0x561f92e469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fee00006010 tx=0x7fee00005e50 comp rx=0 tx=0).stop 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 shutdown_connections 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92e59ba0 0x561f92e5bf90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92e47430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3f140 0x561f92e46ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fedf3400640 0 client.25397 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 --2- 172.21.15.63:0/3023205424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3e820 0x561f92e469b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 >> 172.21.15.63:0/3023205424 conn(0x561f92e3ad30 msgr2=0x561f92e40a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 shutdown_connections 2024-04-25T18:48:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 1 -- 172.21.15.63:0/3023205424 wait complete. 2024-04-25T18:48:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.415+0000 7fee1742b740 20 client.25397 trim_cache size 0 max 16384 2024-04-25T18:48:08.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 14259 13.71 MB/sec warmup 7 sec latency 10.030 ms 2024-04-25T18:48:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.432+0000 7fee1742b740 1 Processor -- start 2024-04-25T18:48:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.432+0000 7fee1742b740 1 -- start start 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.432+0000 7fee1742b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.432+0000 7fee1742b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e39d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee1742b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e49a00 0x561f92e61590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee1742b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1b2640 con 0x561f92e3e820 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee1742b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1aff10 con 0x561f92e3f140 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee1742b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1add60 con 0x561f92e49a00 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51170/0 (socket says 172.21.15.63:51170) 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 -- 172.21.15.63:0/2424964751 learned_addr learned my addr 172.21.15.63:0/2424964751 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11200640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e39d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee12600640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e49a00 0x561f92e61590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 -- 172.21.15.63:0/2424964751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e49a00 msgr2=0x561f92e61590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e49a00 0x561f92e61590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 -- 172.21.15.63:0/2424964751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 msgr2=0x561f92e39d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e39d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11c00640 1 -- 172.21.15.63:0/2424964751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f92e44f00 con 0x561f92e3e820 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.433+0000 7fee11200640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e39d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fee11c00640 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fee00005ab0 tx=0x7fee00005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8ddac72df5c27f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fedfe000640 1 -- 172.21.15.63:0/2424964751 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00053040 con 0x561f92e3e820 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fedfe000640 1 -- 172.21.15.63:0/2424964751 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee00031b10 con 0x561f92e3e820 2024-04-25T18:48:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fedfe000640 1 -- 172.21.15.63:0/2424964751 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00003960 con 0x561f92e3e820 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fee1742b740 1 -- 172.21.15.63:0/2424964751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 msgr2=0x561f92e55490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.434+0000 7fee1742b740 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fee00005ab0 tx=0x7fee00005c00 comp rx=0 tx=0).stop 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 -- 172.21.15.63:0/2424964751 shutdown_connections 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e39d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e49a00 0x561f92e61590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 --2- 172.21.15.63:0/2424964751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e3e820 0x561f92e55490 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 -- 172.21.15.63:0/2424964751 >> 172.21.15.63:0/2424964751 conn(0x561f92e3ad30 msgr2=0x561f92e39600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 -- 172.21.15.63:0/2424964751 shutdown_connections 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 -- 172.21.15.63:0/2424964751 wait complete. 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 -- start start 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.435+0000 7fee1742b740 1 Processor -- start 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 0x561f92e4f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92ebee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1ae3d0 con 0x561f92e49a00 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1af840 con 0x561f92e3f140 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c1aeaa0 con 0x561f92e3e820 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee1742b740 10 client.0 _ll_register_callbacks cb 0x7fee000043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee11200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee11c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 0x561f92e4f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee11200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39994/0 (socket says 172.21.15.63:39994) 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee11200640 1 -- 172.21.15.63:0/982090642 learned_addr learned my addr 172.21.15.63:0/982090642 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.436+0000 7fee12600640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92ebee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 -- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 msgr2=0x561f92e4fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 -- 172.21.15.63:0/982090642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 msgr2=0x561f92ebee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92ebee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 -- 172.21.15.63:0/982090642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f92e50590 con 0x561f92e3e820 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11200640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee12600640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92ebee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee11c00640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 0x561f92e4f750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fee00031940 tx=0x7fee00005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d5a992ed895c189 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fedf3400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00065060 con 0x561f92e3e820 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fee00005390 con 0x561f92e3e820 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee00031b10 con 0x561f92e3e820 2024-04-25T18:48:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee00053070 con 0x561f92e3e820 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee1742b740 10 client.25426 fetch_fsmap learned FSMap version 24 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.437+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561f92ec0490 con 0x561f92e3e820 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee00031b10 con 0x561f92e3e820 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fee00057470 con 0x561f92e3e820 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 10 client.25426 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 10 client.25426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 20 client.25426 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92ec0bb0 0x561f92ec2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 4 client.25426 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f92db2320 con 0x561f92ec0bb0 2024-04-25T18:48:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.438+0000 7fee11200640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92ec0bb0 0x561f92ec2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.439+0000 7fee11200640 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92ec0bb0 0x561f92ec2fd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7039a771f194a7f server_cookie=2fa243614d6312d8 in_seq=0 out_seq=0 2024-04-25T18:48:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.439+0000 7fedf3400640 10 client.25426 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:08.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:08 smithi070 ceph-mon[32820]: pgmap v50: 97 pgs: 97 active+clean; 4.2 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 295 KiB/s wr, 50 op/s 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.531+0000 7fedf3400640 1 -- 172.21.15.63:0/982090642 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f92db2320 con 0x561f92ec0bb0 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.531+0000 7fedf3400640 10 client.25426 handle_command_reply: tid=0 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.531+0000 7fee1742b740 1 client.25426 shutdown 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 msgr2=0x561f92e4f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 0x561f92e4f750 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fee00031940 tx=0x7fee00005e50 comp rx=0 tx=0).stop 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 shutdown_connections 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f92ec0bb0 0x561f92ec2fd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f92e3f140 0x561f92e4fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f92e49a00 0x561f92ebee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fedf3400640 0 client.25426 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 --2- 172.21.15.63:0/982090642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f92e3e820 0x561f92e4f750 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 >> 172.21.15.63:0/982090642 conn(0x561f92e3ad30 msgr2=0x561f92e47200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 shutdown_connections 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 1 -- 172.21.15.63:0/982090642 wait complete. 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.532+0000 7fee1742b740 20 client.25426 trim_cache size 0 max 16384 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:48:08.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 msgr2=0x7fedf40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 0x7fedf40a6280 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fee04005fd0 tx=0x7fee04009600 comp rx=0 tx=0).stop 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 msgr2=0x7fee0c1a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fedf800e9f0 tx=0x7fedf800eeb0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 shutdown_connections 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedf40a3dc0 0x7fedf40a6280 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fee0c109480 0x7fee0c1a0b90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fee0c19a710 0x7fee0c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 --2- 172.21.15.63:0/3948154523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee0c1043b0 0x7fee0c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 >> 172.21.15.63:0/3948154523 conn(0x7fee0c0ffc50 msgr2=0x7fee0c101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 shutdown_connections 2024-04-25T18:48:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:08.533+0000 7fee14600640 1 -- 172.21.15.63:0/3948154523 wait complete. 2024-04-25T18:48:09.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 14918 12.21 MB/sec warmup 8 sec latency 315.522 ms 2024-04-25T18:48:10.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:10 smithi114 ceph-mon[32988]: pgmap v51: 97 pgs: 97 active+clean; 4.6 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 355 KiB/s wr, 70 op/s 2024-04-25T18:48:10.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:10 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:48:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:10 smithi063 ceph-mon[28006]: pgmap v51: 97 pgs: 97 active+clean; 4.6 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 355 KiB/s wr, 70 op/s 2024-04-25T18:48:10.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:10 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:48:10.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 14918 10.85 MB/sec warmup 9 sec latency 1315.688 ms 2024-04-25T18:48:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:10 smithi070 ceph-mon[32820]: pgmap v51: 97 pgs: 97 active+clean; 4.6 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 355 KiB/s wr, 70 op/s 2024-04-25T18:48:10.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:10 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:48:11.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 15844 10.27 MB/sec warmup 10 sec latency 1757.515 ms 2024-04-25T18:48:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:12 smithi063 ceph-mon[28006]: pgmap v52: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 771 KiB/s wr, 176 op/s 2024-04-25T18:48:12.430 INFO:tasks.workunit.client.0.smithi063.stdout: 1 18003 10.57 MB/sec warmup 11 sec latency 13.569 ms 2024-04-25T18:48:12.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:12 smithi070 ceph-mon[32820]: pgmap v52: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 771 KiB/s wr, 176 op/s 2024-04-25T18:48:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:12 smithi114 ceph-mon[32988]: pgmap v52: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 771 KiB/s wr, 176 op/s 2024-04-25T18:48:13.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 18410 9.72 MB/sec warmup 12 sec latency 595.047 ms 2024-04-25T18:48:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:14 smithi063 ceph-mon[28006]: pgmap v53: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 771 KiB/s wr, 173 op/s 2024-04-25T18:48:14.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 18534 9.00 MB/sec warmup 13 sec latency 1183.189 ms 2024-04-25T18:48:14.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:14 smithi070 ceph-mon[32820]: pgmap v53: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 771 KiB/s wr, 173 op/s 2024-04-25T18:48:14.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:14 smithi114 ceph-mon[32988]: pgmap v53: 97 pgs: 97 active+clean; 8.5 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 771 KiB/s wr, 173 op/s 2024-04-25T18:48:15.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 18534 8.35 MB/sec warmup 14 sec latency 1274.208 ms 2024-04-25T18:48:16.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:16 smithi063 ceph-mon[28006]: pgmap v54: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 973 KiB/s wr, 213 op/s 2024-04-25T18:48:16.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 19594 8.13 MB/sec warmup 15 sec latency 1716.043 ms 2024-04-25T18:48:16.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:16 smithi070 ceph-mon[32820]: pgmap v54: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 973 KiB/s wr, 213 op/s 2024-04-25T18:48:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:16 smithi114 ceph-mon[32988]: pgmap v54: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 973 KiB/s wr, 213 op/s 2024-04-25T18:48:17.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 21604 8.46 MB/sec warmup 16 sec latency 11.786 ms 2024-04-25T18:48:18.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:18 smithi063 ceph-mon[28006]: pgmap v55: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 679 KiB/s wr, 165 op/s 2024-04-25T18:48:18.431 INFO:tasks.workunit.client.0.smithi063.stdout: 1 22047 8.00 MB/sec warmup 17 sec latency 545.928 ms 2024-04-25T18:48:18.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:18 smithi070 ceph-mon[32820]: pgmap v55: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 679 KiB/s wr, 165 op/s 2024-04-25T18:48:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:18 smithi114 ceph-mon[32988]: pgmap v55: 97 pgs: 97 active+clean; 10 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 679 KiB/s wr, 165 op/s 2024-04-25T18:48:19.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 22047 7.55 MB/sec warmup 18 sec latency 1546.059 ms 2024-04-25T18:48:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:20 smithi063 ceph-mon[28006]: pgmap v56: 97 pgs: 97 active+clean; 12 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 827 KiB/s wr, 196 op/s 2024-04-25T18:48:20.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 22047 7.16 MB/sec warmup 19 sec latency 2546.178 ms 2024-04-25T18:48:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:20 smithi070 ceph-mon[32820]: pgmap v56: 97 pgs: 97 active+clean; 12 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 827 KiB/s wr, 196 op/s 2024-04-25T18:48:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:20 smithi114 ceph-mon[32988]: pgmap v56: 97 pgs: 97 active+clean; 12 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 827 KiB/s wr, 196 op/s 2024-04-25T18:48:20.736 INFO:tasks.quiescer.fs.[cephfs]:Total duration (46.3) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T18:48:20.737 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.15 2024-04-25T18:48:20.737 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:48:21.220 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:48:21.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 23239 7.06 MB/sec warmup 20 sec latency 2985.231 ms 2024-04-25T18:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.675+0000 7f7bbbe00640 1 -- 172.21.15.63:0/2629418631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074c20 msgr2=0x7f7bbc10e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.675+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/2629418631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074c20 0x7f7bbc10e0f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8009900 tx=0x7f7ba802a480 comp rx=0 tx=0).stop 2024-04-25T18:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 -- 172.21.15.63:0/2629418631 shutdown_connections 2024-04-25T18:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/2629418631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074c20 0x7f7bbc10e0f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/2629418631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 0x7f7bbc110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/2629418631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074300 0x7f7bbc0746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 -- 172.21.15.63:0/2629418631 >> 172.21.15.63:0/2629418631 conn(0x7f7bbc0fd8a0 msgr2=0x7f7bbc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 -- 172.21.15.63:0/2629418631 shutdown_connections 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.676+0000 7f7bbbe00640 1 -- 172.21.15.63:0/2629418631 wait complete. 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 Processor -- start 2024-04-25T18:48:21.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 -- start start 2024-04-25T18:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 0x7f7bbc1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 0x7f7bbc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc114240 con 0x7f7bbc074c20 2024-04-25T18:48:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1140c0 con 0x7f7bbc074300 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1143c0 con 0x7f7bbc10e630 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bba000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 0x7f7bbc1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.677+0000 7f7bba000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38356/0 (socket says 172.21.15.63:38356) 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bba000640 1 -- 172.21.15.63:0/1525326719 learned_addr learned my addr 172.21.15.63:0/1525326719 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbb400640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 0x7f7bbc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 -- 172.21.15.63:0/1525326719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 msgr2=0x7f7bbc19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 0x7f7bbc19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 -- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 msgr2=0x7f7bbc1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 -- 172.21.15.63:0/1525326719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc19ada0 con 0x7f7bbc074300 2024-04-25T18:48:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bba000640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.678+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 0x7f7bbc1a0470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e9f0 tx=0x7f7bac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.679+0000 7f7bc0a00640 1 -- 172.21.15.63:0/1525326719 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac00ccc0 con 0x7f7bbc074300 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.679+0000 7f7bc0a00640 1 -- 172.21.15.63:0/1525326719 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bac0108e0 con 0x7f7bbc074300 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.679+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc19b090 con 0x7f7bbc074300 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.679+0000 7f7bc0a00640 1 -- 172.21.15.63:0/1525326719 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac018880 con 0x7f7bbc074300 2024-04-25T18:48:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.679+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc1a7330 con 0x7f7bbc074300 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.682+0000 7f7bc4bb3740 1 Processor -- start 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.682+0000 7f7bc0a00640 1 -- 172.21.15.63:0/1525326719 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7bac010430 con 0x7f7bbc074300 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.682+0000 7f7bc4bb3740 1 -- start start 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44549b0 0x55cae4454d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 0x55cae445f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cae43ba900 con 0x55cae44549b0 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cae4360ef0 con 0x55cae44552d0 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cae43b68c0 con 0x55cae445fb90 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 0x55cae445f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46324/0 (socket says 172.21.15.63:46324) 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 -- 172.21.15.63:0/2091576040 learned_addr learned my addr 172.21.15.63:0/2091576040 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bbb400640 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44549b0 0x55cae4454d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 -- 172.21.15.63:0/2091576040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 msgr2=0x55cae445f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 0x55cae445f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 -- 172.21.15.63:0/2091576040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44549b0 msgr2=0x55cae4454d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44549b0 0x55cae4454d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bba000640 1 -- 172.21.15.63:0/2091576040 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cae40cd210 con 0x55cae445fb90 2024-04-25T18:48:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.683+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 0x55cae445f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7bba000640 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ba802aa70 tx=0x7f7ba80044b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e23de4c7121c4f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7bc0a00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 0x7f7ba00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7ba7e00640 1 -- 172.21.15.63:0/2091576040 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8040070 con 0x55cae445fb90 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7bc0a00640 1 -- 172.21.15.63:0/1525326719 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7bac0d0ab0 con 0x7f7bbc074300 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7ba7e00640 1 -- 172.21.15.63:0/2091576040 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba80047a0 con 0x55cae445fb90 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7ba7e00640 1 -- 172.21.15.63:0/2091576040 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8035040 con 0x55cae445fb90 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.684+0000 7f7bbb400640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 0x7f7ba00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2091576040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 msgr2=0x55cae4461f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bbb400640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 0x7f7ba00a6250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb4009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7ba802aa70 tx=0x7f7ba80044b0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2091576040 shutdown_connections 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44552d0 0x55cae445f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4461f80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2091576040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44549b0 0x55cae4454d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2091576040 >> 172.21.15.63:0/2091576040 conn(0x55cae4450ec0 msgr2=0x55cae44512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2091576040 shutdown_connections 2024-04-25T18:48:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.685+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2091576040 wait complete. 2024-04-25T18:48:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.686+0000 7f7bc4bb3740 1 -- start start 2024-04-25T18:48:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.686+0000 7f7bc4bb3740 1 Processor -- start 2024-04-25T18:48:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae445d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae445d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cae4462760 con 0x55cae44552d0 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cae4374310 con 0x55cae445fb90 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cae44628e0 con 0x55cae44549b0 2024-04-25T18:48:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bc4bb3740 10 client.0 _ll_register_callbacks cb 0x55cae4342cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46328/0 (socket says 172.21.15.63:46328) 2024-04-25T18:48:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bbaa00640 1 -- 172.21.15.63:0/189839845 learned_addr learned my addr 172.21.15.63:0/189839845 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bbb400640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae445d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.687+0000 7f7bba000640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae445d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 -- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 msgr2=0x55cae445d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae445d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 -- 172.21.15.63:0/189839845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 msgr2=0x55cae445d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae445d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 -- 172.21.15.63:0/189839845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cae4199620 con 0x55cae44549b0 2024-04-25T18:48:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bba000640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae445d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:48:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7bac09ca60 tx=0x7f7bac00ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4541a2969b636c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7ba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac09c430 con 0x55cae44549b0 2024-04-25T18:48:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7bac010a90 con 0x55cae44549b0 2024-04-25T18:48:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba80047a0 con 0x55cae44549b0 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7bac010720 con 0x55cae44549b0 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bc4bb3740 10 client.25438 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.688+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cae446ea10 con 0x55cae44549b0 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ba80047a0 con 0x55cae44549b0 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7bac0bbbe0 con 0x55cae44549b0 2024-04-25T18:48:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 10 client.25438 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 10 client.25438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 20 client.25438 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae446fed0 0x55cae44722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 4 client.25438 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cae43ce8b0 con 0x55cae446fed0 2024-04-25T18:48:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.689+0000 7f7bba000640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae446fed0 0x55cae44722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.690+0000 7f7bba000640 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae446fed0 0x55cae44722e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac5c0fc06eaecd22 server_cookie=a538d952d9a6c003 in_seq=0 out_seq=0 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.690+0000 7f7ba4c00640 10 client.25438 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.690+0000 7f7ba4c00640 1 -- 172.21.15.63:0/189839845 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cae43ce8b0 con 0x55cae446fed0 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.690+0000 7f7ba4c00640 10 client.25438 handle_command_reply: tid=0 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.690+0000 7f7bc4bb3740 1 client.25438 shutdown 2024-04-25T18:48:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 msgr2=0x55cae445cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7bac09ca60 tx=0x7f7bac00ce60 comp rx=0 tx=0).stop 2024-04-25T18:48:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 shutdown_connections 2024-04-25T18:48:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae446fed0 0x55cae44722e0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae445d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae445d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/189839845 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae445cb60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7ba4c00640 0 client.25438 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 >> 172.21.15.63:0/189839845 conn(0x55cae4450ec0 msgr2=0x55cae4456c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 shutdown_connections 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/189839845 wait complete. 2024-04-25T18:48:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.691+0000 7f7bc4bb3740 20 client.25438 trim_cache size 0 max 16384 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.707+0000 7f7bc4bb3740 1 Processor -- start 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.707+0000 7f7bc4bb3740 1 -- start start 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae444fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4477bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1b2620 con 0x55cae44552d0 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1afef0 con 0x55cae44549b0 2024-04-25T18:48:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1add40 con 0x55cae445fb90 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47580/0 (socket says 172.21.15.63:47580) 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 -- 172.21.15.63:0/910127150 learned_addr learned my addr 172.21.15.63:0/910127150 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bba000640 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae444fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbb400640 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4477bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 -- 172.21.15.63:0/910127150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 msgr2=0x55cae4477bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4477bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 -- 172.21.15.63:0/910127150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 msgr2=0x55cae444fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae444fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.708+0000 7f7bbaa00640 1 -- 172.21.15.63:0/910127150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cae444c820 con 0x55cae44549b0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.709+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7bac099450 tx=0x7f7bac098820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1a8d3dabf761a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.709+0000 7f7ba7400640 1 -- 172.21.15.63:0/910127150 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0a5690 con 0x55cae44549b0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.709+0000 7f7ba7400640 1 -- 172.21.15.63:0/910127150 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bac09cad0 con 0x55cae44549b0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.709+0000 7f7ba7400640 1 -- 172.21.15.63:0/910127150 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0989b0 con 0x55cae44549b0 2024-04-25T18:48:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/910127150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 msgr2=0x55cae446b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7bac099450 tx=0x7f7bac098820 comp rx=0 tx=0).stop 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/910127150 shutdown_connections 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae445fb90 0x55cae4477bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae444fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/910127150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae44549b0 0x55cae446b620 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/910127150 >> 172.21.15.63:0/910127150 conn(0x55cae4450ec0 msgr2=0x55cae444f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/910127150 shutdown_connections 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/910127150 wait complete. 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 -- start start 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.710+0000 7f7bc4bb3740 1 Processor -- start 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae44d64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae4464f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1ae3b0 con 0x55cae44552d0 2024-04-25T18:48:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1af820 con 0x55cae445fb90 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1aea80 con 0x55cae44549b0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bc4bb3740 10 client.0 _ll_register_callbacks cb 0x7f7bac09c8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bba000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae44d64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae4464f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46336/0 (socket says 172.21.15.63:46336) 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 -- 172.21.15.63:0/2463982730 learned_addr learned my addr 172.21.15.63:0/2463982730 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 -- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 msgr2=0x55cae4464f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae4464f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 -- 172.21.15.63:0/2463982730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 msgr2=0x55cae44d64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae44d64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.711+0000 7f7bbaa00640 1 -- 172.21.15.63:0/2463982730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cae44656a0 con 0x55cae44549b0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7bbaa00640 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0994a0 tx=0x7f7bac0d4200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba3fd1727eaefc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7ba4c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:21.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac010a90 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7bac0bbdf0 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bac09cad0 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7bac0a5690 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7bc4bb3740 10 client.25444 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.712+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55cae44656a0 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7bac09cad0 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7bac0a2040 con 0x55cae44549b0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 10 client.25444 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 10 client.25444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 20 client.25444 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae44deb10 0x55cae44e0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 4 client.25444 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cae43c8320 con 0x55cae44deb10 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.713+0000 7f7bba000640 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae44deb10 0x55cae44e0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.714+0000 7f7bba000640 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae44deb10 0x55cae44e0f30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=729804ecea78714c server_cookie=48e391b505590eaf in_seq=0 out_seq=0 2024-04-25T18:48:21.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.714+0000 7f7ba4c00640 10 client.25444 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7ba4c00640 1 -- 172.21.15.63:0/2463982730 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cae43c8320 con 0x55cae44deb10 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7ba4c00640 10 client.25444 handle_command_reply: tid=0 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 client.25444 shutdown 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 msgr2=0x55cae44d5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0994a0 tx=0x7f7bac0d4200 comp rx=0 tx=0).stop 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 shutdown_connections 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cae44deb10 0x55cae44e0f30 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cae445fb90 0x55cae4464f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cae44552d0 0x55cae44d64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7ba4c00640 0 client.25444 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:21.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 --2- 172.21.15.63:0/2463982730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cae44549b0 0x55cae44d5f70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.765+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 >> 172.21.15.63:0/2463982730 conn(0x55cae4450ec0 msgr2=0x55cae445d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.766+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 shutdown_connections 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.766+0000 7f7bc4bb3740 1 -- 172.21.15.63:0/2463982730 wait complete. 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.766+0000 7f7bc4bb3740 20 client.25444 trim_cache size 0 max 16384 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:48:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.766+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 msgr2=0x7f7ba00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.766+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 0x7f7ba00a6250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb4009500 comp rx=0 tx=0).stop 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 msgr2=0x7f7bbc1a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 0x7f7bbc1a0470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e9f0 tx=0x7f7bac00eeb0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 shutdown_connections 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7ba00a3d90 0x7f7ba00a6250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7bbc10e630 0x7f7bbc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7bbc074c20 0x7f7bbc1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 --2- 172.21.15.63:0/1525326719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bbc074300 0x7f7bbc1a0470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 >> 172.21.15.63:0/1525326719 conn(0x7f7bbc0fd8a0 msgr2=0x7f7bbc0fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 shutdown_connections 2024-04-25T18:48:21.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:21.767+0000 7f7bbbe00640 1 -- 172.21.15.63:0/1525326719 wait complete. 2024-04-25T18:48:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:22 smithi063 ceph-mon[28006]: pgmap v57: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 817 KiB/s wr, 180 op/s 2024-04-25T18:48:22.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25193 7.36 MB/sec warmup 21 sec latency 24.005 ms 2024-04-25T18:48:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:22 smithi070 ceph-mon[32820]: pgmap v57: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 817 KiB/s wr, 180 op/s 2024-04-25T18:48:22.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:22 smithi114 ceph-mon[32988]: pgmap v57: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 817 KiB/s wr, 180 op/s 2024-04-25T18:48:23.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25541 7.04 MB/sec warmup 22 sec latency 667.910 ms 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 -- 172.21.15.63:0/4033842739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564109c80 msgr2=0x7fe56410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 --2- 172.21.15.63:0/4033842739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564109c80 0x7fe56410dcd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0098f0 tx=0x7fe54c009c00 comp rx=0 tx=0).stop 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 -- 172.21.15.63:0/4033842739 shutdown_connections 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 --2- 172.21.15.63:0/4033842739 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe56410e3b0 0x7fe564115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 --2- 172.21.15.63:0/4033842739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5641092d0 0x7fe5641096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 --2- 172.21.15.63:0/4033842739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe564109c80 0x7fe56410dcd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 -- 172.21.15.63:0/4033842739 >> 172.21.15.63:0/4033842739 conn(0x7fe564076210 msgr2=0x7fe564107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 -- 172.21.15.63:0/4033842739 shutdown_connections 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.974+0000 7fe56b800700 1 -- 172.21.15.63:0/4033842739 wait complete. 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.975+0000 7fe56b800700 1 Processor -- start 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.975+0000 7fe56b800700 1 -- start start 2024-04-25T18:48:24.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564109c80 0x7fe5641a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe56410e3b0 0x7fe56419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe564119270 con 0x7fe56410e3b0 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641190f0 con 0x7fe5641092d0 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe56b800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe5641193f0 con 0x7fe564109c80 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe568e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe568e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47622/0 (socket says 172.21.15.63:47622) 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe568e00700 1 -- 172.21.15.63:0/2570437148 learned_addr learned my addr 172.21.15.63:0/2570437148 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe569800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe56410e3b0 0x7fe56419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.976+0000 7fe55be00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564109c80 0x7fe5641a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 -- 172.21.15.63:0/2570437148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564109c80 msgr2=0x7fe5641a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564109c80 0x7fe5641a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 -- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe56410e3b0 msgr2=0x7fe56419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe56410e3b0 0x7fe56419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 -- 172.21.15.63:0/2570437148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56419fd10 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe568e00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe55400ec10 tx=0x7fe55400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55400cc90 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5641a0000 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe55400ce30 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.977+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554010430 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.978+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641ac2c0 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.979+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe564069730 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.981+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe554004000 con 0x7fe5641092d0 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.982+0000 7fe562000700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 0x7fe5480a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.982+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fe5540d0b70 con 0x7fe5641092d0 2024-04-25T18:48:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.982+0000 7fe55be00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 0x7fe5480a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.983+0000 7fe55be00700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 0x7fe5480a62f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009fd0 tx=0x7fe54c019040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:24.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:23.984+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe55409d1a0 con 0x7fe5641092d0 2024-04-25T18:48:24.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.269+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe56402df80 con 0x7fe5641092d0 2024-04-25T18:48:24.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.270+0000 7fe562000700 1 -- 172.21.15.63:0/2570437148 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7fe55409caa0 con 0x7fe5641092d0 2024-04-25T18:48:24.272 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:48:24.272 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:48:08.989491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 msgr2=0x7fe5480a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 0x7fe5480a62f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009fd0 tx=0x7fe54c019040 comp rx=0 tx=0).stop 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 msgr2=0x7fe5641a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe55400ec10 tx=0x7fe55400ef20 comp rx=0 tx=0).stop 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 shutdown_connections 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe5480a3e30 0x7fe5480a62f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe56410e3b0 0x7fe56419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5641092d0 0x7fe5641a53f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 --2- 172.21.15.63:0/2570437148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe564109c80 0x7fe5641a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.273+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 >> 172.21.15.63:0/2570437148 conn(0x7fe564076210 msgr2=0x7fe5641086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.274+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 shutdown_connections 2024-04-25T18:48:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:24.274+0000 7fe56b800700 1 -- 172.21.15.63:0/2570437148 wait complete. 2024-04-25T18:48:24.276 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 25 2024-04-25T18:48:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:24 smithi063 ceph-mon[28006]: pgmap v58: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 401 KiB/s wr, 74 op/s 2024-04-25T18:48:24.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25541 6.74 MB/sec warmup 23 sec latency 1668.001 ms 2024-04-25T18:48:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:24 smithi070 ceph-mon[32820]: pgmap v58: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 401 KiB/s wr, 74 op/s 2024-04-25T18:48:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:24 smithi114 ceph-mon[32988]: pgmap v58: 97 pgs: 97 active+clean; 12 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 401 KiB/s wr, 74 op/s 2024-04-25T18:48:25.218 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 83.15 --await-for 42.0' 2024-04-25T18:48:25.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 83.15 --await-for 42.0 2024-04-25T18:48:25.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:25 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2570437148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:25.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25541 6.46 MB/sec warmup 24 sec latency 2668.102 ms 2024-04-25T18:48:25.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:25 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2570437148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:25.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:25 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2570437148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:25.685 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:48:26.376 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:26 smithi063 ceph-mon[28006]: pgmap v59: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 579 KiB/s wr, 141 op/s 2024-04-25T18:48:26.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25597 6.20 MB/sec warmup 25 sec latency 3118.030 ms 2024-04-25T18:48:26.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:26 smithi070 ceph-mon[32820]: pgmap v59: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 579 KiB/s wr, 141 op/s 2024-04-25T18:48:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:26 smithi114 ceph-mon[32988]: pgmap v59: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 579 KiB/s wr, 141 op/s 2024-04-25T18:48:27.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25597 5.96 MB/sec warmup 26 sec latency 1483.206 ms 2024-04-25T18:48:28.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:28 smithi063 ceph-mon[28006]: pgmap v60: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 377 KiB/s wr, 101 op/s 2024-04-25T18:48:28.432 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25597 5.74 MB/sec warmup 27 sec latency 2483.306 ms 2024-04-25T18:48:28.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:28 smithi070 ceph-mon[32820]: pgmap v60: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 377 KiB/s wr, 101 op/s 2024-04-25T18:48:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:28 smithi114 ceph-mon[32988]: pgmap v60: 97 pgs: 97 active+clean; 9.6 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 377 KiB/s wr, 101 op/s 2024-04-25T18:48:29.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25597 5.54 MB/sec warmup 28 sec latency 3483.401 ms 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 -- 172.21.15.63:0/2146787599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9001034b0 msgr2=0x7ff90010f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 --2- 172.21.15.63:0/2146787599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9001034b0 0x7ff90010f950 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80098f0 tx=0x7ff8f8009c00 comp rx=0 tx=0).stop 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 -- 172.21.15.63:0/2146787599 shutdown_connections 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 --2- 172.21.15.63:0/2146787599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9001034b0 0x7ff90010f950 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 --2- 172.21.15.63:0/2146787599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 0x7ff900107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 --2- 172.21.15.63:0/2146787599 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff900102af0 0x7ff900102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 -- 172.21.15.63:0/2146787599 >> 172.21.15.63:0/2146787599 conn(0x7ff900079bf0 msgr2=0x7ff90007a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.796+0000 7ff906000700 1 -- 172.21.15.63:0/2146787599 shutdown_connections 2024-04-25T18:48:29.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.797+0000 7ff906000700 1 -- 172.21.15.63:0/2146787599 wait complete. 2024-04-25T18:48:29.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.798+0000 7ff906000700 1 Processor -- start 2024-04-25T18:48:29.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.798+0000 7ff906000700 1 -- start start 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9001034b0 0x7ff900075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 0x7ff9000757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff900112cd0 con 0x7ff9001079c0 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff900112b50 con 0x7ff900102af0 2024-04-25T18:48:29.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff906000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff900112e50 con 0x7ff9001034b0 2024-04-25T18:48:29.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35802/0 (socket says 172.21.15.63:35802) 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff8ff400700 1 -- 172.21.15.63:0/198893413 learned_addr learned my addr 172.21.15.63:0/198893413 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 0x7ff9000757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.799+0000 7ff8fea00700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9001034b0 0x7ff900075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 -- 172.21.15.63:0/198893413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9001034b0 msgr2=0x7ff900075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9001034b0 0x7ff900075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 -- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 msgr2=0x7ff9000757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 0x7ff9000757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 -- 172.21.15.63:0/198893413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9001a76d0 con 0x7ff900102af0 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8ff400700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000e3c0 tx=0x7ff8f000e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8f7e00700 1 -- 172.21.15.63:0/198893413 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000ee40 con 0x7ff900102af0 2024-04-25T18:48:29.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9001a7960 con 0x7ff900102af0 2024-04-25T18:48:29.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8f7e00700 1 -- 172.21.15.63:0/198893413 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f0003a00 con 0x7ff900102af0 2024-04-25T18:48:29.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.800+0000 7ff8f7e00700 1 -- 172.21.15.63:0/198893413 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0004720 con 0x7ff900102af0 2024-04-25T18:48:29.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.801+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9001a7ea0 con 0x7ff900102af0 2024-04-25T18:48:29.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.805+0000 7ff8f7e00700 1 -- 172.21.15.63:0/198893413 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff8f001b030 con 0x7ff900102af0 2024-04-25T18:48:29.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 Processor -- start 2024-04-25T18:48:29.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 -- start start 2024-04-25T18:48:29.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff8f7e00700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 0x7ff8e40a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ac9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9aca180 0x560ea9acc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560ea9a12020 con 0x560ea9abe9b0 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff8f7e00700 1 -- 172.21.15.63:0/198893413 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff8f00d00a0 con 0x7ff900102af0 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560ea99b88c0 con 0x560ea9abf2d0 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff910903b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560ea9a0dfe0 con 0x560ea9aca180 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.807+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 0x7ff8e40a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38586/0 (socket says 172.21.15.63:38586) 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 -- 172.21.15.63:0/790385906 learned_addr learned my addr 172.21.15.63:0/790385906 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8fea00700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ac9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9aca180 0x560ea9acc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 -- 172.21.15.63:0/790385906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9aca180 msgr2=0x560ea9acc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9aca180 0x560ea9acc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 -- 172.21.15.63:0/790385906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 msgr2=0x560ea9ac9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ac9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8ff400700 1 -- 172.21.15.63:0/790385906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea99ae810 con 0x560ea9abe9b0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.808+0000 7ff8fea00700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ac9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.809+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 0x7ff8e40a6310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009fd0 tx=0x7ff8f800fe20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.809+0000 7ff8ff400700 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0000f80 tx=0x7ff8f0098360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66db3867e971468 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.809+0000 7ff8f4c00700 1 -- 172.21.15.63:0/790385906 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f009bc00 con 0x560ea9abe9b0 2024-04-25T18:48:29.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.809+0000 7ff8f4c00700 1 -- 172.21.15.63:0/790385906 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f009c7c0 con 0x560ea9abe9b0 2024-04-25T18:48:29.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.809+0000 7ff8f4c00700 1 -- 172.21.15.63:0/790385906 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f009ba20 con 0x560ea9abe9b0 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 -- 172.21.15.63:0/790385906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 msgr2=0x560ea9abed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0000f80 tx=0x7ff8f0098360 comp rx=0 tx=0).stop 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 -- 172.21.15.63:0/790385906 shutdown_connections 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9aca180 0x560ea9acc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abed90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 --2- 172.21.15.63:0/790385906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ac9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 -- 172.21.15.63:0/790385906 >> 172.21.15.63:0/790385906 conn(0x560ea99ed730 msgr2=0x560ea9abae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 -- 172.21.15.63:0/790385906 shutdown_connections 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.810+0000 7ff910903b80 1 -- 172.21.15.63:0/790385906 wait complete. 2024-04-25T18:48:29.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.811+0000 7ff910903b80 1 -- start start 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.811+0000 7ff910903b80 1 Processor -- start 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9ac98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ac7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560ea9accc40 con 0x560ea9abe9b0 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560ea99cb9f0 con 0x560ea9aca180 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560ea9accdc0 con 0x560ea9abf2d0 2024-04-25T18:48:29.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff910903b80 10 client.0 _ll_register_callbacks cb 0x560ea99aa8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff8fea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff8fea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42602/0 (socket says 172.21.15.63:42602) 2024-04-25T18:48:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff8fea00700 1 -- 172.21.15.63:0/2874780650 learned_addr learned my addr 172.21.15.63:0/2874780650 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ac7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.812+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9ac98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 -- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 msgr2=0x560ea9ac7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ac7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 -- 172.21.15.63:0/2874780650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 msgr2=0x560ea9ac98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9ac98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 -- 172.21.15.63:0/2874780650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea995fa60 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8fea00700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec012980 tx=0x7ff8ec012c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89f5b7629de0c307 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8e9600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec004030 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff8ec0041d0 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.813+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8f009c7c0 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff8ec0043d0 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 10 client.25462 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560ea99a8c80 con 0x560ea9abf2d0 2024-04-25T18:48:29.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f009c7c0 con 0x560ea9abf2d0 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff8ec003a00 con 0x560ea9abf2d0 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 10 client.25462 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 10 client.25462 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 20 client.25462 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9ad9980 0x560ea9adbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 4 client.25462 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.814+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ea9a1f810 con 0x560ea9ad9980 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.815+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9ad9980 0x560ea9adbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.815+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9ad9980 0x560ea9adbd70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66466eb09b0201ac server_cookie=57881cb9f40cdc26 in_seq=0 out_seq=0 2024-04-25T18:48:29.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.815+0000 7ff8e9600700 10 client.25462 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:29.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff8e9600700 1 -- 172.21.15.63:0/2874780650 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ea9a1f810 con 0x560ea9ad9980 2024-04-25T18:48:29.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff8e9600700 10 client.25462 handle_command_reply: tid=0 2024-04-25T18:48:29.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 client.25462 shutdown 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 msgr2=0x560ea9ac6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec012980 tx=0x7ff8ec012c90 comp rx=0 tx=0).stop 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 shutdown_connections 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9ad9980 0x560ea9adbd70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ac7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff8e9600700 0 client.25462 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9ac98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.816+0000 7ff910903b80 1 --2- 172.21.15.63:0/2874780650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac6bc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.817+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 >> 172.21.15.63:0/2874780650 conn(0x560ea99ed730 msgr2=0x560ea9abaa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.817+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 shutdown_connections 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.817+0000 7ff910903b80 1 -- 172.21.15.63:0/2874780650 wait complete. 2024-04-25T18:48:29.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.817+0000 7ff910903b80 20 client.25462 trim_cache size 0 max 16384 2024-04-25T18:48:29.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.839+0000 7ff910903b80 1 Processor -- start 2024-04-25T18:48:29.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.839+0000 7ff910903b80 1 -- start start 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ae1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001b1ed0 con 0x560ea9abe9b0 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001af760 con 0x560ea9aca180 2024-04-25T18:48:29.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff910903b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001ad5b0 con 0x560ea9abf2d0 2024-04-25T18:48:29.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff8ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38612/0 (socket says 172.21.15.63:38612) 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff8ff400700 1 -- 172.21.15.63:0/2182876510 learned_addr learned my addr 172.21.15.63:0/2182876510 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ae1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.840+0000 7ff8fea00700 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 -- 172.21.15.63:0/2182876510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 msgr2=0x560ea9ac21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 -- 172.21.15.63:0/2182876510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 msgr2=0x560ea9ae1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ae1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 -- 172.21.15.63:0/2182876510 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea9adc3d0 con 0x560ea9abe9b0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00cd840 tx=0x7ff8f00041c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2ea06901938bb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ebe00700 1 -- 172.21.15.63:0/2182876510 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00b5070 con 0x560ea9abe9b0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ebe00700 1 -- 172.21.15.63:0/2182876510 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8f009c0c0 con 0x560ea9abe9b0 2024-04-25T18:48:29.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.841+0000 7ff8ebe00700 1 -- 172.21.15.63:0/2182876510 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f00a0470 con 0x560ea9abe9b0 2024-04-25T18:48:29.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 -- 172.21.15.63:0/2182876510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 msgr2=0x560ea9abafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff8f00cd840 tx=0x7ff8f00041c0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 -- 172.21.15.63:0/2182876510 shutdown_connections 2024-04-25T18:48:29.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9aca180 0x560ea9ae1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9abe9b0 0x560ea9abafe0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 --2- 172.21.15.63:0/2182876510 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abf2d0 0x560ea9ac21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 -- 172.21.15.63:0/2182876510 >> 172.21.15.63:0/2182876510 conn(0x560ea99ed730 msgr2=0x560ea9ac1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 -- 172.21.15.63:0/2182876510 shutdown_connections 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.842+0000 7ff910903b80 1 -- 172.21.15.63:0/2182876510 wait complete. 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.843+0000 7ff910903b80 1 -- start start 2024-04-25T18:48:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.843+0000 7ff910903b80 1 Processor -- start 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abe9b0 0x560ea9ad2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ace8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001adc20 con 0x560ea9aca180 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001af090 con 0x560ea9abf2d0 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9001ae2f0 con 0x560ea9abe9b0 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff910903b80 10 client.0 _ll_register_callbacks cb 0x560ea99aa8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff8ffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff8fea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ace8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff8ffe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38624/0 (socket says 172.21.15.63:38624) 2024-04-25T18:48:29.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff8ffe00700 1 -- 172.21.15.63:0/2875182377 learned_addr learned my addr 172.21.15.63:0/2875182377 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.844+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abe9b0 0x560ea9ad2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 -- 172.21.15.63:0/2875182377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abe9b0 msgr2=0x560ea9ad2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abe9b0 0x560ea9ad2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 -- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 msgr2=0x560ea9ace8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ace8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 -- 172.21.15.63:0/2875182377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea9abf6b0 con 0x560ea9aca180 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8fea00700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ace8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8ffe00700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80098f0 tx=0x7ff8f8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edfc0076bf1b9fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8e9600700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8020040 con 0x560ea9aca180 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff8f8005500 con 0x560ea9aca180 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.845+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560ea9a198b0 con 0x560ea9aca180 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff8f8040050 con 0x560ea9aca180 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 10 client.15291 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:29.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560ea9b40e10 con 0x560ea9aca180 2024-04-25T18:48:29.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560ea9a198b0 con 0x560ea9aca180 2024-04-25T18:48:29.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff8f800b760 con 0x560ea9aca180 2024-04-25T18:48:29.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 10 client.15291 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:29.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 10 client.15291 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 20 client.15291 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9b41670 0x560ea9b43a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 4 client.15291 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.15, "await_for": 42.0}] 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.846+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.15, "await_for": 42.0}) -- 0x560ea98923e0 con 0x560ea9b41670 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.847+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9b41670 0x560ea9b43a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.847+0000 7ff8ff400700 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9b41670 0x560ea9b43a90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d4ff8dfa584183 server_cookie=72365bb254a4967f in_seq=0 out_seq=0 2024-04-25T18:48:29.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.847+0000 7ff8e9600700 10 client.15291 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:29.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.886+0000 7ff8e9600700 1 -- 172.21.15.63:0/2875182377 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x560ea98923e0 con 0x560ea9b41670 2024-04-25T18:48:29.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.886+0000 7ff8e9600700 10 client.15291 handle_command_reply: tid=0 2024-04-25T18:48:29.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 client.15291 shutdown 2024-04-25T18:48:29.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 msgr2=0x560ea9acf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80098f0 tx=0x7ff8f8009c00 comp rx=0 tx=0).stop 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 shutdown_connections 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560ea9b41670 0x560ea9b43a90 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560ea9abe9b0 0x560ea9ad2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ea9aca180 0x560ea9acf020 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 --2- 172.21.15.63:0/2875182377 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560ea9abf2d0 0x560ea9ace8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 >> 172.21.15.63:0/2875182377 conn(0x560ea99ed730 msgr2=0x560ea9ac5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.887+0000 7ff8e9600700 0 client.15291 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.888+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 shutdown_connections 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.888+0000 7ff910903b80 1 -- 172.21.15.63:0/2875182377 wait complete. 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.888+0000 7ff910903b80 20 client.15291 trim_cache size 0 max 16384 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 25, 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 6, 2024-04-25T18:48:29.890 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "c672114c": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 6, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 83.2, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:48:29.891 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:48:29.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 msgr2=0x7ff8e40a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 0x7ff8e40a6310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009fd0 tx=0x7ff8f800fe20 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 msgr2=0x7ff900074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000e3c0 tx=0x7ff8f000e6d0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 shutdown_connections 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff8e40a3e50 0x7ff8e40a6310 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9001079c0 0x7ff9000757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff900102af0 0x7ff900074d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 --2- 172.21.15.63:0/198893413 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9001034b0 0x7ff900075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 >> 172.21.15.63:0/198893413 conn(0x7ff900079bf0 msgr2=0x7ff900100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 shutdown_connections 2024-04-25T18:48:29.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:29.889+0000 7ff906000700 1 -- 172.21.15.63:0/198893413 wait complete. 2024-04-25T18:48:30.305 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:30 smithi063 ceph-mon[28006]: pgmap v61: 97 pgs: 97 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-04-25T18:48:30.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 5.35 MB/sec warmup 29 sec latency 3937.924 ms 2024-04-25T18:48:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:30 smithi070 ceph-mon[32820]: pgmap v61: 97 pgs: 97 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-04-25T18:48:30.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:30 smithi114 ceph-mon[32988]: pgmap v61: 97 pgs: 97 active+clean; 21 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 1.3 MiB/s wr, 111 op/s 2024-04-25T18:48:30.922 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c672114c', quiesce took 10.2 seconds. Will release after: 23.15 2024-04-25T18:48:31.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 5.17 MB/sec warmup 30 sec latency 1545.716 ms 2024-04-25T18:48:32.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 5.00 MB/sec warmup 31 sec latency 2545.818 ms 2024-04-25T18:48:32.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:32 smithi070 ceph-mon[32820]: pgmap v62: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.9 MiB/s wr, 84 op/s 2024-04-25T18:48:32.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:32 smithi114 ceph-mon[32988]: pgmap v62: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.9 MiB/s wr, 84 op/s 2024-04-25T18:48:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:32 smithi063 ceph-mon[28006]: pgmap v62: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.9 MiB/s wr, 84 op/s 2024-04-25T18:48:33.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.85 MB/sec warmup 32 sec latency 3545.966 ms 2024-04-25T18:48:34.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.70 MB/sec warmup 33 sec latency 4546.058 ms 2024-04-25T18:48:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:34 smithi070 ceph-mon[32820]: pgmap v63: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-04-25T18:48:34.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:34 smithi114 ceph-mon[32988]: pgmap v63: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-04-25T18:48:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:34 smithi063 ceph-mon[28006]: pgmap v63: 97 pgs: 97 active+clean; 29 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 -- 172.21.15.63:0/1375227547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108560 msgr2=0x7f43901089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 --2- 172.21.15.63:0/1375227547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108560 0x7f43901089e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f437c009930 tx=0x7f437c02a440 comp rx=0 tx=0).stop 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 -- 172.21.15.63:0/1375227547 shutdown_connections 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 --2- 172.21.15.63:0/1375227547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108560 0x7f43901089e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 --2- 172.21.15.63:0/1375227547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4390108f20 0x7f43901153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 --2- 172.21.15.63:0/1375227547 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f439010e620 0x7f439010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.921+0000 7f4396800640 1 -- 172.21.15.63:0/1375227547 >> 172.21.15.63:0/1375227547 conn(0x7f43900764e0 msgr2=0x7f4390078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.922+0000 7f4396800640 1 -- 172.21.15.63:0/1375227547 shutdown_connections 2024-04-25T18:48:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.922+0000 7f4396800640 1 -- 172.21.15.63:0/1375227547 wait complete. 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 Processor -- start 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 -- start start 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108f20 0x7f43901a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 0x7f439019ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f43901187e0 con 0x7f439010e620 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4390118660 con 0x7f4390108f20 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f4396800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4390118960 con 0x7f4390108560 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f438fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 0x7f439019ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42648/0 (socket says 172.21.15.63:42648) 2024-04-25T18:48:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.923+0000 7f438f400640 1 -- 172.21.15.63:0/925053628 learned_addr learned my addr 172.21.15.63:0/925053628 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438ea00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108f20 0x7f43901a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 -- 172.21.15.63:0/925053628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 msgr2=0x7f43901a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 -- 172.21.15.63:0/925053628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108f20 msgr2=0x7f43901a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108f20 0x7f43901a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 -- 172.21.15.63:0/925053628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439019f200 con 0x7f439010e620 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438f400640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.924+0000 7f438fe00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 0x7f439019ea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f438800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.925+0000 7f4387e00640 1 -- 172.21.15.63:0/925053628 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438800ce20 con 0x7f439010e620 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.925+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439019f4f0 con 0x7f439010e620 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.925+0000 7f4387e00640 1 -- 172.21.15.63:0/925053628 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4388005020 con 0x7f439010e620 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.925+0000 7f4387e00640 1 -- 172.21.15.63:0/925053628 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388010690 con 0x7f439010e620 2024-04-25T18:48:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.925+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43901ab790 con 0x7f439010e620 2024-04-25T18:48:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.928+0000 7f4399768740 1 Processor -- start 2024-04-25T18:48:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.928+0000 7f4387e00640 1 -- 172.21.15.63:0/925053628 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f43880054d0 con 0x7f439010e620 2024-04-25T18:48:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.928+0000 7f4399768740 1 -- start start 2024-04-25T18:48:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.928+0000 7f4399768740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563816410820 0x563816410c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f4399768740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f4399768740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56381641ba00 0x56381641ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f4399768740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563816376900 con 0x56381641ba00 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f4399768740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56381631cef0 con 0x563816410820 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f4399768740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5638163728c0 con 0x563816411140 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563816410820 0x563816410c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42656/0 (socket says 172.21.15.63:42656) 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 -- 172.21.15.63:0/1680889522 learned_addr learned my addr 172.21.15.63:0/1680889522 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438ea00640 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56381641ba00 0x56381641ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 -- 172.21.15.63:0/1680889522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563816410820 msgr2=0x563816410c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563816410820 0x563816410c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 -- 172.21.15.63:0/1680889522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56381641ba00 msgr2=0x56381641ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56381641ba00 0x56381641ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.929+0000 7f438f400640 1 -- 172.21.15.63:0/1680889522 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563816089210 con 0x563816411140 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f4387e00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 0x7f43740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f438f400640 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f43800115f0 tx=0x7f4380011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=185db8c180a2c714 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f4387e00640 1 -- 172.21.15.63:0/925053628 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f43880d0ef0 con 0x7f439010e620 2024-04-25T18:48:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f438ea00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 0x7f43740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f4384c00640 1 -- 172.21.15.63:0/1680889522 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438001e070 con 0x563816411140 2024-04-25T18:48:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f4384c00640 1 -- 172.21.15.63:0/1680889522 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f438000f660 con 0x563816411140 2024-04-25T18:48:34.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.930+0000 7f4384c00640 1 -- 172.21.15.63:0/1680889522 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380003850 con 0x563816411140 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f438ea00640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 0x7f43740a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f437c031440 tx=0x7f437c005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 -- 172.21.15.63:0/1680889522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 msgr2=0x56381641b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f43800115f0 tx=0x7f4380011ab0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 -- 172.21.15.63:0/1680889522 shutdown_connections 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56381641ba00 0x56381641ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x56381641b4c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 --2- 172.21.15.63:0/1680889522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563816410820 0x563816410c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 -- 172.21.15.63:0/1680889522 >> 172.21.15.63:0/1680889522 conn(0x56381640cd30 msgr2=0x56381640d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 -- 172.21.15.63:0/1680889522 shutdown_connections 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.931+0000 7f4399768740 1 -- 172.21.15.63:0/1680889522 wait complete. 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 -- start start 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 Processor -- start 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 0x563816418ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816419410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56381641e630 con 0x563816411140 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563816330310 con 0x56381641ba00 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.932+0000 7f4399768740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56381641e7b0 con 0x563816410820 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f4399768740 10 client.0 _ll_register_callbacks cb 0x5638162fecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438ea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 0x563816418ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438ea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 0x563816418ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38670/0 (socket says 172.21.15.63:38670) 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42668/0 (socket says 172.21.15.63:42668) 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438ea00640 1 -- 172.21.15.63:0/2768132571 learned_addr learned my addr 172.21.15.63:0/2768132571 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438fe00640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816419410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 -- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 msgr2=0x563816419410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816419410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 -- 172.21.15.63:0/2768132571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 msgr2=0x563816418ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 0x563816418ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.933+0000 7f438f400640 1 -- 172.21.15.63:0/2768132571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563816155620 con 0x563816410820 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f438f400640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f43800115f0 tx=0x7f43800052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6f5a01bf407e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4379600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380026020 con 0x563816410820 2024-04-25T18:48:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f43800054e0 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f438000f660 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4380031050 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4399768740 10 client.25480 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.934+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56381642a890 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f438000f660 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4380021030 con 0x563816410820 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 10 client.25480 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 20 client.25480 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56381642bc90 0x56381642e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 4 client.25480 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56381638a8b0 con 0x56381642bc90 2024-04-25T18:48:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.935+0000 7f438ea00640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56381642bc90 0x56381642e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.936+0000 7f438ea00640 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56381642bc90 0x56381642e080 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d25f2d5f83c6578 server_cookie=9a61012243a1655f in_seq=0 out_seq=0 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.936+0000 7f4379600640 10 client.25480 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4379600640 1 -- 172.21.15.63:0/2768132571 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56381638a8b0 con 0x56381642bc90 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4379600640 10 client.25480 handle_command_reply: tid=0 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 client.25480 shutdown 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 msgr2=0x563816418990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f43800115f0 tx=0x7f43800052f0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 shutdown_connections 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56381642bc90 0x56381642e080 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816419410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816411140 0x563816418ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4379600640 0 client.25480 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 --2- 172.21.15.63:0/2768132571 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816410820 0x563816418990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 >> 172.21.15.63:0/2768132571 conn(0x56381640cd30 msgr2=0x563816412a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 shutdown_connections 2024-04-25T18:48:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 1 -- 172.21.15.63:0/2768132571 wait complete. 2024-04-25T18:48:34.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.937+0000 7f4399768740 20 client.25480 trim_cache size 0 max 16384 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.954+0000 7f4399768740 1 Processor -- start 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.954+0000 7f4399768740 1 -- start start 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x5638164274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816418270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x5638164121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b69a0 con 0x563816410820 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b4270 con 0x56381641ba00 2024-04-25T18:48:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f4399768740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b20c0 con 0x563816411140 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x5638164274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816418270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x5638164274f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38678/0 (socket says 172.21.15.63:38678) 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438f400640 1 -- 172.21.15.63:0/1835971521 learned_addr learned my addr 172.21.15.63:0/1835971521 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438fe00640 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x5638164121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 -- 172.21.15.63:0/1835971521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 msgr2=0x5638164121d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x5638164121d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 -- 172.21.15.63:0/1835971521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 msgr2=0x5638164274f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x5638164274f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.955+0000 7f438ea00640 1 -- 172.21.15.63:0/1835971521 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56381640d900 con 0x563816411140 2024-04-25T18:48:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f438ea00640 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816418270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f437c006010 tx=0x7f437c03f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f429c8ddf5e176c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f437be00640 1 -- 172.21.15.63:0/1835971521 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c05a070 con 0x563816411140 2024-04-25T18:48:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f437be00640 1 -- 172.21.15.63:0/1835971521 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f437c054580 con 0x563816411140 2024-04-25T18:48:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f437be00640 1 -- 172.21.15.63:0/1835971521 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c031ae0 con 0x563816411140 2024-04-25T18:48:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f4399768740 1 -- 172.21.15.63:0/1835971521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 msgr2=0x563816418270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.956+0000 7f4399768740 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816418270 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f437c006010 tx=0x7f437c03f040 comp rx=0 tx=0).stop 2024-04-25T18:48:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 -- 172.21.15.63:0/1835971521 shutdown_connections 2024-04-25T18:48:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x5638164121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816418270 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 --2- 172.21.15.63:0/1835971521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x5638164274f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 -- 172.21.15.63:0/1835971521 >> 172.21.15.63:0/1835971521 conn(0x56381640cd30 msgr2=0x5638164191e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 -- 172.21.15.63:0/1835971521 shutdown_connections 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 -- 172.21.15.63:0/1835971521 wait complete. 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 -- start start 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.957+0000 7f4399768740 1 Processor -- start 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816422680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816422bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b2730 con 0x563816410820 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b3ba0 con 0x56381641ba00 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43901b2e00 con 0x563816411140 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f4399768740 10 client.0 _ll_register_callbacks cb 0x7f438000fe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38680/0 (socket says 172.21.15.63:38680) 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438f400640 1 -- 172.21.15.63:0/2383829249 learned_addr learned my addr 172.21.15.63:0/2383829249 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438ea00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816422680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438fe00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816422bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438f400640 1 -- 172.21.15.63:0/2383829249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 msgr2=0x563816422680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.958+0000 7f438f400640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816422680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438f400640 1 -- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 msgr2=0x563816422bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438f400640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816422bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438f400640 1 -- 172.21.15.63:0/2383829249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563816423130 con 0x563816410820 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438fe00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816422bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438ea00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816422680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f438f400640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4380000c00 tx=0x7f438000fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b93cd42fb02431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4379600640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380026040 con 0x563816410820 2024-04-25T18:48:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f437c054580 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f4380005360 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f438001e040 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4399768740 10 client.15309 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.959+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563816491ec0 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f437c054580 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4380021070 con 0x563816410820 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 10 client.15309 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 10 client.15309 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 20 client.15309 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563816492520 0x563816494940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 4 client.15309 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.960+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563816384320 con 0x563816492520 2024-04-25T18:48:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.961+0000 7f438ea00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563816492520 0x563816494940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.961+0000 7f438ea00640 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563816492520 0x563816494940 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e2ab0ed30e224c1 server_cookie=5f9a3e5b90808342 in_seq=0 out_seq=0 2024-04-25T18:48:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:34.961+0000 7f4379600640 10 client.15309 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4379600640 1 -- 172.21.15.63:0/2383829249 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563816384320 con 0x563816492520 2024-04-25T18:48:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4379600640 10 client.15309 handle_command_reply: tid=0 2024-04-25T18:48:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 client.15309 shutdown 2024-04-25T18:48:35.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 msgr2=0x563816423b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4380000c00 tx=0x7f438000fd50 comp rx=0 tx=0).stop 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 shutdown_connections 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563816492520 0x563816494940 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56381641ba00 0x563816422bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563816411140 0x563816422680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4379600640 0 client.15309 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 --2- 172.21.15.63:0/2383829249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563816410820 0x563816423b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.001+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 >> 172.21.15.63:0/2383829249 conn(0x56381640cd30 msgr2=0x563816418880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.002+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 shutdown_connections 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.002+0000 7f4399768740 1 -- 172.21.15.63:0/2383829249 wait complete. 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.002+0000 7f4399768740 20 client.15309 trim_cache size 0 max 16384 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:48:35.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 msgr2=0x7f43740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 0x7f43740a6300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f437c031440 tx=0x7f437c005a20 comp rx=0 tx=0).stop 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 msgr2=0x7f439019ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 0x7f439019ea80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4388009fd0 tx=0x7f438800ef10 comp rx=0 tx=0).stop 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 shutdown_connections 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f43740a3e40 0x7f43740a6300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4390108f20 0x7f43901a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f439010e620 0x7f439019ea80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 --2- 172.21.15.63:0/925053628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4390108560 0x7f43901a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 >> 172.21.15.63:0/925053628 conn(0x7f43900764e0 msgr2=0x7f4390077dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 shutdown_connections 2024-04-25T18:48:35.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:35.003+0000 7f4396800640 1 -- 172.21.15.63:0/925053628 wait complete. 2024-04-25T18:48:35.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.56 MB/sec warmup 34 sec latency 5546.218 ms 2024-04-25T18:48:36.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.43 MB/sec warmup 35 sec latency 6546.351 ms 2024-04-25T18:48:36.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:36 smithi070 ceph-mon[32820]: pgmap v64: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.5 MiB/s wr, 115 op/s 2024-04-25T18:48:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:36 smithi114 ceph-mon[32988]: pgmap v64: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.5 MiB/s wr, 115 op/s 2024-04-25T18:48:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:36 smithi063 ceph-mon[28006]: pgmap v64: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 3.5 MiB/s wr, 115 op/s 2024-04-25T18:48:37.433 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.31 MB/sec warmup 36 sec latency 7546.446 ms 2024-04-25T18:48:38.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.19 MB/sec warmup 37 sec latency 8546.586 ms 2024-04-25T18:48:38.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:38 smithi070 ceph-mon[32820]: pgmap v65: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-04-25T18:48:38.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:38 smithi114 ceph-mon[32988]: pgmap v65: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-04-25T18:48:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:38 smithi063 ceph-mon[28006]: pgmap v65: 97 pgs: 97 active+clean; 48 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 3.3 MiB/s wr, 49 op/s 2024-04-25T18:48:39.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 4.08 MB/sec warmup 38 sec latency 9546.769 ms 2024-04-25T18:48:40.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.98 MB/sec warmup 39 sec latency 10546.912 ms 2024-04-25T18:48:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:40 smithi070 ceph-mon[32820]: pgmap v66: 97 pgs: 97 active+clean; 34 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 3.4 MiB/s wr, 74 op/s 2024-04-25T18:48:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:40 smithi114 ceph-mon[32988]: pgmap v66: 97 pgs: 97 active+clean; 34 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 3.4 MiB/s wr, 74 op/s 2024-04-25T18:48:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:40 smithi063 ceph-mon[28006]: pgmap v66: 97 pgs: 97 active+clean; 34 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 3.4 MiB/s wr, 74 op/s 2024-04-25T18:48:41.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.88 MB/sec warmup 40 sec latency 11547.006 ms 2024-04-25T18:48:42.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.78 MB/sec warmup 41 sec latency 12547.161 ms 2024-04-25T18:48:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:42 smithi070 ceph-mon[32820]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 78 op/s 2024-04-25T18:48:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:42 smithi114 ceph-mon[32988]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 78 op/s 2024-04-25T18:48:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:42 smithi063 ceph-mon[28006]: pgmap v67: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 2.4 MiB/s wr, 78 op/s 2024-04-25T18:48:43.434 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.69 MB/sec warmup 42 sec latency 13547.317 ms 2024-04-25T18:48:44.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.61 MB/sec warmup 43 sec latency 14547.402 ms 2024-04-25T18:48:44.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:44 smithi070 ceph-mon[32820]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-04-25T18:48:44.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:44 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:44.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:44 smithi114 ceph-mon[32988]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-04-25T18:48:44.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:44 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:44 smithi063 ceph-mon[28006]: pgmap v68: 97 pgs: 97 active+clean; 26 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.7 MiB/s wr, 73 op/s 2024-04-25T18:48:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:44 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:48:45.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.52 MB/sec warmup 44 sec latency 15547.522 ms 2024-04-25T18:48:45.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:45 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:45.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:45 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:45.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:45 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:48:45.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:45 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:45.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:45 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:45.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:45 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:48:45.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:45 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:48:45.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:45 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:48:45.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:45 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:48:46.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.45 MB/sec warmup 45 sec latency 16547.658 ms 2024-04-25T18:48:46.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:46 smithi070 ceph-mon[32820]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-25T18:48:46.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:46 smithi114 ceph-mon[32988]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-25T18:48:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:46 smithi063 ceph-mon[28006]: pgmap v69: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-04-25T18:48:47.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.37 MB/sec warmup 46 sec latency 17547.818 ms 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 -- 172.21.15.63:0/298814453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f01043a0 msgr2=0x7f73f0104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 --2- 172.21.15.63:0/298814453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f01043a0 0x7f73f0104780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009900 tx=0x7f73e002a3f0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 -- 172.21.15.63:0/298814453 shutdown_connections 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 --2- 172.21.15.63:0/298814453 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f0109470 0x7f73f0111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 --2- 172.21.15.63:0/298814453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0104d50 0x7f73f0108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 --2- 172.21.15.63:0/298814453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f01043a0 0x7f73f0104780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 -- 172.21.15.63:0/298814453 >> 172.21.15.63:0/298814453 conn(0x7f73f00ffc40 msgr2=0x7f73f0102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 -- 172.21.15.63:0/298814453 shutdown_connections 2024-04-25T18:48:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.154+0000 7f73f5c00640 1 -- 172.21.15.63:0/298814453 wait complete. 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 Processor -- start 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 -- start start 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0109470 0x7f73f01a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 0x7f73f019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0114400 con 0x7f73f0109470 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0114280 con 0x7f73f019a6a0 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.156+0000 7f73f5c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73f0114580 con 0x7f73f01043a0 2024-04-25T18:48:48.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0109470 0x7f73f01a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54358/0 (socket says 172.21.15.63:54358) 2024-04-25T18:48:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73eea00640 1 -- 172.21.15.63:0/2652463821 learned_addr learned my addr 172.21.15.63:0/2652463821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 0x7f73f019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 -- 172.21.15.63:0/2652463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 msgr2=0x7f73f01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 -- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0109470 msgr2=0x7f73f01a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0109470 0x7f73f01a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73ef400640 1 -- 172.21.15.63:0/2652463821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f019b2a0 con 0x7f73f019a6a0 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.157+0000 7f73eea00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73ef400640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 0x7f73f019ab20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009fd0 tx=0x7f73e800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73e7400640 1 -- 172.21.15.63:0/2652463821 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800ce20 con 0x7f73f019a6a0 2024-04-25T18:48:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73e7400640 1 -- 172.21.15.63:0/2652463821 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e8005020 con 0x7f73f019a6a0 2024-04-25T18:48:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f01a7380 con 0x7f73f019a6a0 2024-04-25T18:48:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73e7400640 1 -- 172.21.15.63:0/2652463821 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8010690 con 0x7f73f019a6a0 2024-04-25T18:48:48.161 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.158+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f01a7780 con 0x7f73f019a6a0 2024-04-25T18:48:48.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.161+0000 7f73f8bd6740 1 Processor -- start 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73e7400640 1 -- 172.21.15.63:0/2652463821 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f73e80054d0 con 0x7f73f019a6a0 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 -- start start 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625fbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625fe600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564d6254e900 con 0x564d625f1030 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564d624f4ef0 con 0x564d625f1950 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564d6254a8c0 con 0x564d625fc210 2024-04-25T18:48:48.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56954/0 (socket says 172.21.15.63:56954) 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73e7e00640 1 -- 172.21.15.63:0/3841548289 learned_addr learned my addr 172.21.15.63:0/3841548289 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.162+0000 7f73eea00640 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625fbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625fe600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 -- 172.21.15.63:0/3841548289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 msgr2=0x564d625fe600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625fe600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 -- 172.21.15.63:0/3841548289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 msgr2=0x564d625fbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625fbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 -- 172.21.15.63:0/3841548289 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d62261210 con 0x564d625f1030 2024-04-25T18:48:48.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73e7e00640 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f73d80135c0 tx=0x7f73d80099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de4b68b8d877d52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73dfe00640 1 -- 172.21.15.63:0/3841548289 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8004050 con 0x564d625f1030 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73dfe00640 1 -- 172.21.15.63:0/3841548289 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73d80041f0 con 0x564d625f1030 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.163+0000 7f73dfe00640 1 -- 172.21.15.63:0/3841548289 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d800f760 con 0x564d625f1030 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73e7400640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 0x7f73d40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 -- 172.21.15.63:0/3841548289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 msgr2=0x564d625f1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f73d80135c0 tx=0x7f73d80099d0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 -- 172.21.15.63:0/3841548289 shutdown_connections 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625fbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625fe600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/3841548289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f1410 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73f8bd6740 1 -- 172.21.15.63:0/3841548289 >> 172.21.15.63:0/3841548289 conn(0x564d625ed540 msgr2=0x564d625ed920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73e7400640 1 -- 172.21.15.63:0/2652463821 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f73e80d0e30 con 0x7f73f019a6a0 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.164+0000 7f73eea00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 0x7f73d40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.165+0000 7f73f8bd6740 1 -- 172.21.15.63:0/3841548289 shutdown_connections 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.165+0000 7f73f8bd6740 1 -- 172.21.15.63:0/3841548289 wait complete. 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.165+0000 7f73eea00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 0x7f73d40a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x564d625f25d0 tx=0x7f73e0005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.165+0000 7f73f8bd6740 1 -- start start 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.165+0000 7f73f8bd6740 1 Processor -- start 2024-04-25T18:48:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564d625fee40 con 0x564d625f1030 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564d62508310 con 0x564d625f1950 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564d625fefc0 con 0x564d625fc210 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73f8bd6740 10 client.0 _ll_register_callbacks cb 0x564d624d6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73eea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73eea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56966/0 (socket says 172.21.15.63:56966) 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73eea00640 1 -- 172.21.15.63:0/1176688895 learned_addr learned my addr 172.21.15.63:0/1176688895 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73ef400640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.166+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 -- 172.21.15.63:0/1176688895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 msgr2=0x564d625f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 -- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 msgr2=0x564d625f9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 -- 172.21.15.63:0/1176688895 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d6232d620 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73eea00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f73e002a9e0 tx=0x7f73e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2ae3fead1276c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0035040 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f73e0031a30 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.167+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73d80041f0 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f73e0031c30 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73f8bd6740 10 client.15321 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d6260b020 con 0x564d625f1030 2024-04-25T18:48:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73d80041f0 con 0x564d625f1030 2024-04-25T18:48:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f73e0044030 con 0x564d625f1030 2024-04-25T18:48:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73f8bd6740 10 client.15321 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.168+0000 7f73f8bd6740 10 client.15321 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73f8bd6740 20 client.15321 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d6260c530 0x564d6260e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73f8bd6740 4 client.15321 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d625628b0 con 0x564d6260c530 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d6260c530 0x564d6260e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d6260c530 0x564d6260e9c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2998fe9550b2c73 server_cookie=914e41350bff175 in_seq=0 out_seq=0 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.169+0000 7f73dcc00640 10 client.15321 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.170+0000 7f73dcc00640 1 -- 172.21.15.63:0/1176688895 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d625628b0 con 0x564d6260c530 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.170+0000 7f73dcc00640 10 client.15321 handle_command_reply: tid=0 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.170+0000 7f73f8bd6740 1 client.15321 shutdown 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 msgr2=0x564d625f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f73e002a9e0 tx=0x7f73e0009260 comp rx=0 tx=0).stop 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 shutdown_connections 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d6260c530 0x564d6260e9c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d625f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73dcc00640 0 client.15321 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1176688895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1030 0x564d625f90f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 >> 172.21.15.63:0/1176688895 conn(0x564d625ed540 msgr2=0x564d625e3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 shutdown_connections 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1176688895 wait complete. 2024-04-25T18:48:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.171+0000 7f73f8bd6740 20 client.15321 trim_cache size 0 max 16384 2024-04-25T18:48:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 Processor -- start 2024-04-25T18:48:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 -- start start 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625fc210 0x564d62614290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01ae120 con 0x564d625fc210 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01ab9f0 con 0x564d625f1950 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.188+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01a9840 con 0x564d625f1030 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54390/0 (socket says 172.21.15.63:54390) 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 -- 172.21.15.63:0/2353987166 learned_addr learned my addr 172.21.15.63:0/2353987166 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73e7e00640 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73ef400640 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625fc210 0x564d62614290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 -- 172.21.15.63:0/2353987166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 msgr2=0x564d625f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 -- 172.21.15.63:0/2353987166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625fc210 msgr2=0x564d62614290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625fc210 0x564d62614290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.189+0000 7f73eea00640 1 -- 172.21.15.63:0/2353987166 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d625ed370 con 0x564d625f1030 2024-04-25T18:48:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.190+0000 7f73eea00640 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73e0000c00 tx=0x7f73e002a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3622f5c009dd0454 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.190+0000 7f73df400640 1 -- 172.21.15.63:0/2353987166 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e005e030 con 0x564d625f1030 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.190+0000 7f73df400640 1 -- 172.21.15.63:0/2353987166 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e004c070 con 0x564d625f1030 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.190+0000 7f73df400640 1 -- 172.21.15.63:0/2353987166 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e0051920 con 0x564d625f1030 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- 172.21.15.63:0/2353987166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 msgr2=0x564d62607d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f73e0000c00 tx=0x7f73e002a3f0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- 172.21.15.63:0/2353987166 shutdown_connections 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1950 0x564d625f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625fc210 0x564d62614290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/2353987166 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625f1030 0x564d62607d00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- 172.21.15.63:0/2353987166 >> 172.21.15.63:0/2353987166 conn(0x564d625ed540 msgr2=0x564d625e3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- 172.21.15.63:0/2353987166 shutdown_connections 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- 172.21.15.63:0/2353987166 wait complete. 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.191+0000 7f73f8bd6740 1 -- start start 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 Processor -- start 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1030 0x564d62604620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d62601ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01a9eb0 con 0x564d625f1950 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01ab320 con 0x564d625f1030 2024-04-25T18:48:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01aa580 con 0x564d625fc210 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73f8bd6740 10 client.0 _ll_register_callbacks cb 0x7f73e00037d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56982/0 (socket says 172.21.15.63:56982) 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73e7e00640 1 -- 172.21.15.63:0/1015935406 learned_addr learned my addr 172.21.15.63:0/1015935406 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73ef400640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d62601ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.192+0000 7f73eea00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1030 0x564d62604620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 -- 172.21.15.63:0/1015935406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 msgr2=0x564d62601ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d62601ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 -- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1030 msgr2=0x564d62604620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1030 0x564d62604620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 -- 172.21.15.63:0/1015935406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d62602720 con 0x564d625f1950 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73ef400640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d62601ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73e7e00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009fd0 tx=0x7f73d8005c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b841591ab73def server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8005240 con 0x564d625f1950 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f73d8005040 con 0x564d625f1950 2024-04-25T18:48:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.193+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73e004c070 con 0x564d625f1950 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f73d80053e0 con 0x564d625f1950 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 10 client.15327 fetch_fsmap learned FSMap version 25 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564d62672b80 con 0x564d625f1950 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e004c070 con 0x564d625f1950 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f73d80039d0 con 0x564d625f1950 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 10 client.15327 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 10 client.15327 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 20 client.15327 populate_metadata read hostname 'smithi063' 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d62673030 0x564d62675450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 4 client.15327 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.194+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d6255c320 con 0x564d62673030 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.195+0000 7f73eea00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d62673030 0x564d62675450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.195+0000 7f73eea00640 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d62673030 0x564d62675450 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baf6e4b06267397d server_cookie=dcbfa1b0df25c316 in_seq=0 out_seq=0 2024-04-25T18:48:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.195+0000 7f73dcc00640 10 client.15327 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73dcc00640 1 -- 172.21.15.63:0/1015935406 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d6255c320 con 0x564d62673030 2024-04-25T18:48:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73dcc00640 10 client.15327 handle_command_reply: tid=0 2024-04-25T18:48:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 client.15327 shutdown 2024-04-25T18:48:48.214 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 msgr2=0x564d626019b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009fd0 tx=0x7f73d8005c10 comp rx=0 tx=0).stop 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 shutdown_connections 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564d62673030 0x564d62675450 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564d625fc210 0x564d62601ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d625f1950 0x564d626019b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 --2- 172.21.15.63:0/1015935406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564d625f1030 0x564d62604620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 >> 172.21.15.63:0/1015935406 conn(0x564d625ed540 msgr2=0x564d625f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73dcc00640 0 client.15327 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 shutdown_connections 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 1 -- 172.21.15.63:0/1015935406 wait complete. 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.213+0000 7f73f8bd6740 20 client.15327 trim_cache size 0 max 16384 2024-04-25T18:48:48.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:48:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:48:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:48:48.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 msgr2=0x7f73d40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 0x7f73d40a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x564d625f25d0 tx=0x7f73e0005a20 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 msgr2=0x7f73f019ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 0x7f73f019ab20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009fd0 tx=0x7f73e800ef10 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 shutdown_connections 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73d40a3e40 0x7f73d40a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73f019a6a0 0x7f73f019ab20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73f0109470 0x7f73f01a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 --2- 172.21.15.63:0/2652463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f73f01043a0 0x7f73f01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.214+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 >> 172.21.15.63:0/2652463821 conn(0x7f73f00ffc40 msgr2=0x7f73f0101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.215+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 shutdown_connections 2024-04-25T18:48:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:48:48.215+0000 7f73f5c00640 1 -- 172.21.15.63:0/2652463821 wait complete. 2024-04-25T18:48:48.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.30 MB/sec warmup 47 sec latency 18547.967 ms 2024-04-25T18:48:48.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:48 smithi070 ceph-mon[32820]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 89 KiB/s wr, 42 op/s 2024-04-25T18:48:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:48 smithi114 ceph-mon[32988]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 89 KiB/s wr, 42 op/s 2024-04-25T18:48:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:48 smithi063 ceph-mon[28006]: pgmap v70: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 89 KiB/s wr, 42 op/s 2024-04-25T18:48:49.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.23 MB/sec warmup 48 sec latency 19548.073 ms 2024-04-25T18:48:50.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.17 MB/sec warmup 49 sec latency 20548.196 ms 2024-04-25T18:48:50.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:50 smithi070 ceph-mon[32820]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 96 KiB/s wr, 48 op/s 2024-04-25T18:48:50.466 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:50 smithi063 ceph-mon[28006]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 96 KiB/s wr, 48 op/s 2024-04-25T18:48:50.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:50 smithi114 ceph-mon[32988]: pgmap v71: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 96 KiB/s wr, 48 op/s 2024-04-25T18:48:51.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.10 MB/sec warmup 50 sec latency 21548.284 ms 2024-04-25T18:48:52.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:52 smithi063 ceph-mon[28006]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 23 op/s 2024-04-25T18:48:52.435 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 3.04 MB/sec warmup 51 sec latency 22548.425 ms 2024-04-25T18:48:52.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:52 smithi070 ceph-mon[32820]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 23 op/s 2024-04-25T18:48:52.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:52 smithi114 ceph-mon[32988]: pgmap v72: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 KiB/s wr, 23 op/s 2024-04-25T18:48:53.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.98 MB/sec warmup 52 sec latency 23548.577 ms 2024-04-25T18:48:54.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:48:54.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.93 MB/sec warmup 53 sec latency 24548.686 ms 2024-04-25T18:48:54.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:54 smithi070 ceph-mon[32820]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T18:48:54.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:54 smithi114 ceph-mon[32988]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T18:48:54.568 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:48:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:54 smithi063 ceph-mon[28006]: pgmap v73: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-25T18:48:55.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.87 MB/sec warmup 54 sec latency 25548.755 ms 2024-04-25T18:48:56.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.82 MB/sec warmup 55 sec latency 26548.842 ms 2024-04-25T18:48:56.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:56 smithi070 ceph-mon[32820]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T18:48:56.547 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:56 smithi063 ceph-mon[28006]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T18:48:56.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:56 smithi114 ceph-mon[32988]: pgmap v74: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-04-25T18:48:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 -- 172.21.15.63:0/2835948994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 msgr2=0x7ff93010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 --2- 172.21.15.63:0/2835948994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff93010dcd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff9180099c0 tx=0x7ff918009cd0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 -- 172.21.15.63:0/2835948994 shutdown_connections 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 --2- 172.21.15.63:0/2835948994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff93010e3b0 0x7ff930115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 --2- 172.21.15.63:0/2835948994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9301092d0 0x7ff9301096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 --2- 172.21.15.63:0/2835948994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff93010dcd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 -- 172.21.15.63:0/2835948994 >> 172.21.15.63:0/2835948994 conn(0x7ff930076210 msgr2=0x7ff930107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 -- 172.21.15.63:0/2835948994 shutdown_connections 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.912+0000 7ff937000700 1 -- 172.21.15.63:0/2835948994 wait complete. 2024-04-25T18:48:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 Processor -- start 2024-04-25T18:48:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 -- start start 2024-04-25T18:48:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff9301a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff93010e3b0 0x7ff93019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff930119270 con 0x7ff930109c80 2024-04-25T18:48:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9301190f0 con 0x7ff93010e3b0 2024-04-25T18:48:56.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.914+0000 7ff937000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff9301193f0 con 0x7ff9301092d0 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff92fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff9301a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff92fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54414/0 (socket says 172.21.15.63:54414) 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff92fe00700 1 -- 172.21.15.63:0/2160305250 learned_addr learned my addr 172.21.15.63:0/2160305250 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff935000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff93010e3b0 0x7ff93019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 -- 172.21.15.63:0/2160305250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 msgr2=0x7ff9301a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 -- 172.21.15.63:0/2160305250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff93010e3b0 msgr2=0x7ff93019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff93010e3b0 0x7ff93019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff927400700 1 -- 172.21.15.63:0/2160305250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93019fd50 con 0x7ff930109c80 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.915+0000 7ff92fe00700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:48:56.917 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff927400700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff9301a5970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9180098e0 tx=0x7ff918005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:56.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918026070 con 0x7ff930109c80 2024-04-25T18:48:56.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9180056c0 con 0x7ff930109c80 2024-04-25T18:48:56.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91800f960 con 0x7ff930109c80 2024-04-25T18:48:56.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93019ffe0 con 0x7ff930109c80 2024-04-25T18:48:56.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.916+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9301ac300 con 0x7ff930109c80 2024-04-25T18:48:56.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.917+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff930069730 con 0x7ff930109c80 2024-04-25T18:48:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.921+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff918029070 con 0x7ff930109c80 2024-04-25T18:48:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.922+0000 7ff92d600700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 0x7ff9140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:48:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.923+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff9180d1340 con 0x7ff930109c80 2024-04-25T18:48:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.923+0000 7ff92fe00700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 0x7ff9140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:48:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.924+0000 7ff92fe00700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 0x7ff9140a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff920011630 tx=0x7ff920009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:48:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:56.927+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff918030020 con 0x7ff930109c80 2024-04-25T18:48:57.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.216+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff93002df80 con 0x7ff930109c80 2024-04-25T18:48:57.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.217+0000 7ff92d600700 1 -- 172.21.15.63:0/2160305250 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7ff918013070 con 0x7ff930109c80 2024-04-25T18:48:57.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:48:57.219 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:48:08.989491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 msgr2=0x7ff9140a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 0x7ff9140a62f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff920011630 tx=0x7ff920009260 comp rx=0 tx=0).stop 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 msgr2=0x7ff9301a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff9301a5970 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff9180098e0 tx=0x7ff918005280 comp rx=0 tx=0).stop 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 shutdown_connections 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff9140a3e30 0x7ff9140a62f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff9301092d0 0x7ff9301a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff93010e3b0 0x7ff93019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:57.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 --2- 172.21.15.63:0/2160305250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff930109c80 0x7ff9301a5970 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:48:57.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.220+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 >> 172.21.15.63:0/2160305250 conn(0x7ff930076210 msgr2=0x7ff930077e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:48:57.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.221+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 shutdown_connections 2024-04-25T18:48:57.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:48:57.221+0000 7ff937000700 1 -- 172.21.15.63:0/2160305250 wait complete. 2024-04-25T18:48:57.223 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 25 2024-04-25T18:48:57.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.77 MB/sec warmup 56 sec latency 27548.931 ms 2024-04-25T18:48:58.171 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id c672114c --release --await' 2024-04-25T18:48:58.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id c672114c --release --await 2024-04-25T18:48:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:58 smithi063 ceph-mon[28006]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T18:48:58.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:48:58 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2160305250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:58.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.72 MB/sec warmup 57 sec latency 28549.057 ms 2024-04-25T18:48:58.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:58 smithi114 ceph-mon[32988]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T18:48:58.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:48:58 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2160305250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:58.643 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:48:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:58 smithi070 ceph-mon[32820]: pgmap v75: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 7.2 KiB/s wr, 9 op/s 2024-04-25T18:48:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:48:58 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2160305250' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:48:59.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.67 MB/sec warmup 58 sec latency 29549.154 ms 2024-04-25T18:49:00.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 25598 2.63 MB/sec warmup 59 sec latency 30549.311 ms 2024-04-25T18:49:00.471 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:00 smithi063 ceph-mon[28006]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.8 KiB/s wr, 11 op/s 2024-04-25T18:49:00.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:00 smithi114 ceph-mon[32988]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.8 KiB/s wr, 11 op/s 2024-04-25T18:49:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:00 smithi070 ceph-mon[32820]: pgmap v76: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.8 KiB/s wr, 11 op/s 2024-04-25T18:49:01.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 -- 172.21.15.63:0/3411593060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 msgr2=0x7f9d50102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/3411593060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d50102df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009990 tx=0x7f9d3c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 -- 172.21.15.63:0/3411593060 shutdown_connections 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/3411593060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d5010f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/3411593060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 0x7f9d50108d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/3411593060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d50102df0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 -- 172.21.15.63:0/3411593060 >> 172.21.15.63:0/3411593060 conn(0x7f9d500fe4c0 msgr2=0x7f9d501008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 -- 172.21.15.63:0/3411593060 shutdown_connections 2024-04-25T18:49:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.053+0000 7f9d4fe00700 1 -- 172.21.15.63:0/3411593060 wait complete. 2024-04-25T18:49:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.054+0000 7f9d4fe00700 1 Processor -- start 2024-04-25T18:49:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.054+0000 7f9d4fe00700 1 -- start start 2024-04-25T18:49:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d501a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d501a5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 0x7f9d5010d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50112bd0 con 0x7f9d50108970 2024-04-25T18:49:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50112a50 con 0x7f9d50103330 2024-04-25T18:49:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4fe00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50112d50 con 0x7f9d50102970 2024-04-25T18:49:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.055+0000 7f9d4ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d501a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d501a5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56042/0 (socket says 172.21.15.63:56042) 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4ea00700 1 -- 172.21.15.63:0/1693137698 learned_addr learned my addr 172.21.15.63:0/1693137698 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d501a5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4f400700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 0x7f9d5010d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 -- 172.21.15.63:0/1693137698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 msgr2=0x7f9d501a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d501a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 -- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 msgr2=0x7f9d5010d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 0x7f9d5010d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.056+0000 7f9d4e000700 1 -- 172.21.15.63:0/1693137698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5010e070 con 0x7f9d50103330 2024-04-25T18:49:01.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d4e000700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d501a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c000c00 tx=0x7f9d3c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d47400700 1 -- 172.21.15.63:0/1693137698 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c026070 con 0x7f9d50103330 2024-04-25T18:49:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5010e300 con 0x7f9d50103330 2024-04-25T18:49:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d47400700 1 -- 172.21.15.63:0/1693137698 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c021410 con 0x7f9d50103330 2024-04-25T18:49:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d47400700 1 -- 172.21.15.63:0/1693137698 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c017650 con 0x7f9d50103330 2024-04-25T18:49:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.057+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d501ac190 con 0x7f9d50103330 2024-04-25T18:49:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.061+0000 7f9d47400700 1 -- 172.21.15.63:0/1693137698 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d3c029070 con 0x7f9d50103330 2024-04-25T18:49:01.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.063+0000 7f9d5ffb8b80 1 Processor -- start 2024-04-25T18:49:01.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.063+0000 7f9d5ffb8b80 1 -- start start 2024-04-25T18:49:01.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f5a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f65130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d47400700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 0x7f9d340a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bf1eacf90 con 0x561bf1f5a7c0 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bf1e53830 con 0x561bf1f59ea0 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bf1ea8f50 con 0x561bf1f65670 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d47400700 1 -- 172.21.15.63:0/1693137698 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d3c0d16f0 con 0x7f9d50103330 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d4f400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d4f400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56044/0 (socket says 172.21.15.63:56044) 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d4f400700 1 -- 172.21.15.63:0/2059309176 learned_addr learned my addr 172.21.15.63:0/2059309176 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.064+0000 7f9d4f400700 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f5a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 -- 172.21.15.63:0/2059309176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 msgr2=0x561bf1f5a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f5a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 -- 172.21.15.63:0/2059309176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 msgr2=0x561bf1f65130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f65130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 -- 172.21.15.63:0/2059309176 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bf1e49780 con 0x561bf1f65670 2024-04-25T18:49:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 0x7f9d340a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d4f400700 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9d48012df0 tx=0x7f9d48009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31cade423460bb73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d3be00700 1 -- 172.21.15.63:0/2059309176 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48009310 con 0x561bf1f65670 2024-04-25T18:49:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d3be00700 1 -- 172.21.15.63:0/2059309176 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d4801f070 con 0x561bf1f65670 2024-04-25T18:49:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.065+0000 7f9d3be00700 1 -- 172.21.15.63:0/2059309176 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48007c50 con 0x561bf1f65670 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.066+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/2059309176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 msgr2=0x561bf1f67a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.066+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 0x7f9d340a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d40007c10 tx=0x7f9d40007480 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.066+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9d48012df0 tx=0x7f9d48009f90 comp rx=0 tx=0).stop 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/2059309176 shutdown_connections 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f67a60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f5a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/2059309176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f65130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/2059309176 >> 172.21.15.63:0/2059309176 conn(0x561bf1e89050 msgr2=0x561bf1f563f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/2059309176 shutdown_connections 2024-04-25T18:49:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/2059309176 wait complete. 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.067+0000 7f9d5ffb8b80 1 -- start start 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 Processor -- start 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f64fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bf1f79110 con 0x561bf1f59ea0 2024-04-25T18:49:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bf1e66960 con 0x561bf1f65670 2024-04-25T18:49:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bf1f79290 con 0x561bf1f5a7c0 2024-04-25T18:49:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.068+0000 7f9d5ffb8b80 10 client.0 _ll_register_callbacks cb 0x561bf1e45820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57838/0 (socket says 172.21.15.63:57838) 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 -- 172.21.15.63:0/1830072414 learned_addr learned my addr 172.21.15.63:0/1830072414 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f64fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4e000700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 -- 172.21.15.63:0/1830072414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 msgr2=0x561bf1f622a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f622a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 -- 172.21.15.63:0/1830072414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 msgr2=0x561bf1f64fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f64fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.069+0000 7f9d4f400700 1 -- 172.21.15.63:0/1830072414 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bf1dfa9d0 con 0x561bf1f65670 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d4f400700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d48007810 tx=0x7f9d48005b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf44e5d98bb2cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d38c00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48017d50 con 0x561bf1f65670 2024-04-25T18:49:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d48007910 con 0x561bf1f65670 2024-04-25T18:49:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d4801f070 con 0x561bf1f65670 2024-04-25T18:49:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.070+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d48007b10 con 0x561bf1f65670 2024-04-25T18:49:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 10 client.25499 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561bf1e43bf0 con 0x561bf1f65670 2024-04-25T18:49:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d4801f070 con 0x561bf1f65670 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9d4801ab90 con 0x561bf1f65670 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 10 client.25499 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 10 client.25499 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 20 client.25499 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1f85ed0 0x561bf1f882c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 4 client.25499 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.071+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bf1eba780 con 0x561bf1f85ed0 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.072+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1f85ed0 0x561bf1f882c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.072+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1f85ed0 0x561bf1f882c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d48d9331e35379d server_cookie=9efcd89bd7f0dfb8 in_seq=0 out_seq=0 2024-04-25T18:49:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.072+0000 7f9d38c00700 10 client.25499 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.073+0000 7f9d38c00700 1 -- 172.21.15.63:0/1830072414 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bf1eba780 con 0x561bf1f85ed0 2024-04-25T18:49:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.073+0000 7f9d38c00700 10 client.25499 handle_command_reply: tid=0 2024-04-25T18:49:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.073+0000 7f9d5ffb8b80 1 client.25499 shutdown 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.073+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 msgr2=0x561bf1f62800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d48007810 tx=0x7f9d48005b90 comp rx=0 tx=0).stop 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 shutdown_connections 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1f85ed0 0x561bf1f882c0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f62800 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f64fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/1830072414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f622a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 >> 172.21.15.63:0/1830072414 conn(0x561bf1e89050 msgr2=0x561bf1f55fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d38c00700 0 client.25499 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 shutdown_connections 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/1830072414 wait complete. 2024-04-25T18:49:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.074+0000 7f9d5ffb8b80 20 client.25499 trim_cache size 0 max 16384 2024-04-25T18:49:01.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.101+0000 7f9d5ffb8b80 1 Processor -- start 2024-04-25T18:49:01.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.101+0000 7f9d5ffb8b80 1 -- start start 2024-04-25T18:49:01.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.101+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.101+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f8dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f5f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501ad860 con 0x561bf1f5a7c0 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50073940 con 0x561bf1f59ea0 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50071790 con 0x561bf1f65670 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57852/0 (socket says 172.21.15.63:57852) 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 -- 172.21.15.63:0/441233093 learned_addr learned my addr 172.21.15.63:0/441233093 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4f400700 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f5f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 -- 172.21.15.63:0/441233093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 msgr2=0x561bf1f5f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f5f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 -- 172.21.15.63:0/441233093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 msgr2=0x561bf1f8dec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f8dec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.102+0000 7f9d4ea00700 1 -- 172.21.15.63:0/441233093 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bf1f5b060 con 0x561bf1f59ea0 2024-04-25T18:49:01.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.103+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d40010d20 tx=0x7f9d40013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0ae669c311464c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.103+0000 7f9d3b400700 1 -- 172.21.15.63:0/441233093 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4002e070 con 0x561bf1f59ea0 2024-04-25T18:49:01.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.103+0000 7f9d3b400700 1 -- 172.21.15.63:0/441233093 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40011db0 con 0x561bf1f59ea0 2024-04-25T18:49:01.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.103+0000 7f9d3b400700 1 -- 172.21.15.63:0/441233093 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4001fd60 con 0x561bf1f59ea0 2024-04-25T18:49:01.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/441233093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 msgr2=0x561bf1f8d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d40010d20 tx=0x7f9d40013000 comp rx=0 tx=0).stop 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/441233093 shutdown_connections 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f65670 0x561bf1f5f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f59ea0 0x561bf1f8d980 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/441233093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f5a7c0 0x561bf1f8dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/441233093 >> 172.21.15.63:0/441233093 conn(0x561bf1e89050 msgr2=0x561bf1f62260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.104+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/441233093 shutdown_connections 2024-04-25T18:49:01.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.105+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/441233093 wait complete. 2024-04-25T18:49:01.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.105+0000 7f9d5ffb8b80 1 -- start start 2024-04-25T18:49:01.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.105+0000 7f9d5ffb8b80 1 Processor -- start 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f7e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f7cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50071e00 con 0x561bf1f59ea0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50073270 con 0x561bf1f65670 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d500724d0 con 0x561bf1f5a7c0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d5ffb8b80 10 client.0 _ll_register_callbacks cb 0x561bf1e45820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d4e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d4e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56060/0 (socket says 172.21.15.63:56060) 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d4ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f7e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d4e000700 1 -- 172.21.15.63:0/3910356308 learned_addr learned my addr 172.21.15.63:0/3910356308 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.106+0000 7f9d4f400700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f7cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 -- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 msgr2=0x561bf1f7cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f7cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 -- 172.21.15.63:0/3910356308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 msgr2=0x561bf1f7e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f7e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 -- 172.21.15.63:0/3910356308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bf1f7d510 con 0x561bf1f5a7c0 2024-04-25T18:49:01.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:01.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d4e000700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c09a080 tx=0x7f9d3c099950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c13b83fb4fd678c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d38c00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:01.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c09d010 con 0x561bf1f5a7c0 2024-04-25T18:49:01.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9d3c00fb10 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d40011db0 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.107+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9d3c09cb00 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 10 client.25528 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561bf1fed050 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40011db0 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9d3c0218c0 con 0x561bf1f5a7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 10 client.25528 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 10 client.25528 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 20 client.25528 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1fed7c0 0x561bf1fefbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 4 client.25528 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c672114c", "release": true, "await": true}] 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c672114c", "release": true, "await": true}) -- 0x561bf1eb4820 con 0x561bf1fed7c0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.108+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1fed7c0 0x561bf1fefbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.109+0000 7f9d4ea00700 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1fed7c0 0x561bf1fefbe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c05aa43ffc6b0f server_cookie=48050fd86893ae8e in_seq=0 out_seq=0 2024-04-25T18:49:01.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.109+0000 7f9d38c00700 10 client.25528 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.113+0000 7f9d38c00700 1 -- 172.21.15.63:0/3910356308 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x561bf1eb4820 con 0x561bf1fed7c0 2024-04-25T18:49:01.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.113+0000 7f9d38c00700 10 client.25528 handle_command_reply: tid=0 2024-04-25T18:49:01.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.113+0000 7f9d5ffb8b80 1 client.25528 shutdown 2024-04-25T18:49:01.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 msgr2=0x561bf1f7c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c09a080 tx=0x7f9d3c099950 comp rx=0 tx=0).stop 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 shutdown_connections 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bf1fed7c0 0x561bf1fefbe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bf1f65670 0x561bf1f7cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bf1f59ea0 0x561bf1f7e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d38c00700 0 client.25528 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 --2- 172.21.15.63:0/3910356308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bf1f5a7c0 0x561bf1f7c9e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 >> 172.21.15.63:0/3910356308 conn(0x561bf1e89050 msgr2=0x561bf1f60810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 shutdown_connections 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 1 -- 172.21.15.63:0/3910356308 wait complete. 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.114+0000 7f9d5ffb8b80 20 client.25528 trim_cache size 0 max 16384 2024-04-25T18:49:01.116 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 25, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 8, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "c672114c": { 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 8, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 83.2, 2024-04-25T18:49:01.117 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:01.118 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 msgr2=0x7f9d340a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 0x7f9d340a62f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d40007c10 tx=0x7f9d40007480 comp rx=0 tx=0).stop 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 msgr2=0x7f9d501a5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d501a5870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c000c00 tx=0x7f9d3c0047b0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 shutdown_connections 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e30 0x7f9d340a62f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d50102970 0x7f9d501a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50108970 0x7f9d5010d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 --2- 172.21.15.63:0/1693137698 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50103330 0x7f9d501a5870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 >> 172.21.15.63:0/1693137698 conn(0x7f9d500fe4c0 msgr2=0x7f9d50075150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 shutdown_connections 2024-04-25T18:49:01.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:01.115+0000 7f9d4fe00700 1 -- 172.21.15.63:0/1693137698 wait complete. 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 -- 172.21.15.63:0/2783763901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b010e610 msgr2=0x7f95b0110a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 --2- 172.21.15.63:0/2783763901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b010e610 0x7f95b0110a20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a802a480 comp rx=0 tx=0).stop 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 -- 172.21.15.63:0/2783763901 shutdown_connections 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 --2- 172.21.15.63:0/2783763901 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b010e610 0x7f95b0110a20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 --2- 172.21.15.63:0/2783763901 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 0x7f95b010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 --2- 172.21.15.63:0/2783763901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b0074320 0x7f95b0074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 -- 172.21.15.63:0/2783763901 >> 172.21.15.63:0/2783763901 conn(0x7f95b00fd8a0 msgr2=0x7f95b00ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.369+0000 7f95afe00640 1 -- 172.21.15.63:0/2783763901 shutdown_connections 2024-04-25T18:49:01.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 -- 172.21.15.63:0/2783763901 wait complete. 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 Processor -- start 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 -- start start 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 0x7f95b01a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 0x7f95b019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95b0114250 con 0x7f95b0074c40 2024-04-25T18:49:01.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01140d0 con 0x7f95b010e610 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.370+0000 7f95afe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01143d0 con 0x7f95b0074320 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 0x7f95b019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56082/0 (socket says 172.21.15.63:56082) 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 -- 172.21.15.63:0/39404258 learned_addr learned my addr 172.21.15.63:0/39404258 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95a6000640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 0x7f95b01a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 -- 172.21.15.63:0/39404258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 msgr2=0x7f95b019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 0x7f95b019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 -- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 msgr2=0x7f95b01a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 0x7f95b01a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95aea00640 1 -- 172.21.15.63:0/39404258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b019adb0 con 0x7f95b0074320 2024-04-25T18:49:01.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.371+0000 7f95af400640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 0x7f95b019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95aea00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f95a000eb90 tx=0x7f95a000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95a7e00640 1 -- 172.21.15.63:0/39404258 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0004d20 con 0x7f95b0074320 2024-04-25T18:49:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95a7e00640 1 -- 172.21.15.63:0/39404258 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a000ee90 con 0x7f95b0074320 2024-04-25T18:49:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95a7e00640 1 -- 172.21.15.63:0/39404258 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a0010670 con 0x7f95b0074320 2024-04-25T18:49:01.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b019b0a0 con 0x7f95b0074320 2024-04-25T18:49:01.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.372+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b01a72f0 con 0x7f95b0074320 2024-04-25T18:49:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.376+0000 7f95a7e00640 1 -- 172.21.15.63:0/39404258 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f95a0005020 con 0x7f95b0074320 2024-04-25T18:49:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.376+0000 7f95b860d740 1 Processor -- start 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 -- start start 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b959650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56539b8b4900 con 0x56539b94f2d0 2024-04-25T18:49:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56539b85aef0 con 0x56539b94e9b0 2024-04-25T18:49:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.377+0000 7f95b860d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56539b8b08c0 con 0x56539b959b90 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56090/0 (socket says 172.21.15.63:56090) 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 -- 172.21.15.63:0/453180073 learned_addr learned my addr 172.21.15.63:0/453180073 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57886/0 (socket says 172.21.15.63:57886) 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95aea00640 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b959650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 -- 172.21.15.63:0/453180073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 msgr2=0x56539b94ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 -- 172.21.15.63:0/453180073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 msgr2=0x56539b959650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b959650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 -- 172.21.15.63:0/453180073 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539b5c7210 con 0x56539b959b90 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95af400640 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a6000640 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9598009b70 tx=0x7f95980117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5766ee2aefd5002 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.378+0000 7f95a7e00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 0x7f95940a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.379+0000 7f95a7e00640 1 -- 172.21.15.63:0/39404258 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f95a00d0e30 con 0x7f95b0074320 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.379+0000 7f959fe00640 1 -- 172.21.15.63:0/453180073 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9598009e60 con 0x56539b959b90 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.379+0000 7f959fe00640 1 -- 172.21.15.63:0/453180073 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9598012900 con 0x56539b959b90 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.379+0000 7f959fe00640 1 -- 172.21.15.63:0/453180073 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959800fa50 con 0x56539b959b90 2024-04-25T18:49:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.379+0000 7f95af400640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 0x7f95940a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 -- 172.21.15.63:0/453180073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 msgr2=0x56539b95bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9598009b70 tx=0x7f95980117b0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95af400640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 0x7f95940a6250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95a802a9a0 tx=0x7f95a8005a40 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 -- 172.21.15.63:0/453180073 shutdown_connections 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b95bf80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b959650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 --2- 172.21.15.63:0/453180073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b94ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 -- 172.21.15.63:0/453180073 >> 172.21.15.63:0/453180073 conn(0x56539b94aec0 msgr2=0x56539b94b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 -- 172.21.15.63:0/453180073 shutdown_connections 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.380+0000 7f95b860d740 1 -- 172.21.15.63:0/453180073 wait complete. 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 -- start start 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 Processor -- start 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b956b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b957070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b9575b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56539b95c7c0 con 0x56539b94f2d0 2024-04-25T18:49:01.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56539b86e310 con 0x56539b94e9b0 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.381+0000 7f95b860d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56539b95c940 con 0x56539b959b90 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95b860d740 10 client.0 _ll_register_callbacks cb 0x56539b83ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b9575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b956b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95af400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b9575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56094/0 (socket says 172.21.15.63:56094) 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95af400640 1 -- 172.21.15.63:0/3302618859 learned_addr learned my addr 172.21.15.63:0/3302618859 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95a6000640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b957070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 -- 172.21.15.63:0/3302618859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 msgr2=0x56539b9575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b9575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 -- 172.21.15.63:0/3302618859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 msgr2=0x56539b957070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b957070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.382+0000 7f95aea00640 1 -- 172.21.15.63:0/3302618859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539b693620 con 0x56539b94e9b0 2024-04-25T18:49:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f95aea00640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b956b30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f95a00ce570 tx=0x7f95a00d1030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55fded2acbc4ee3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f959cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a009c900 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f95a00b62c0 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9598012900 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f95a009c720 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f95b860d740 10 client.25517 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.383+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56539b968aa0 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.384+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9598012900 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f95a00d4b30 con 0x56539b94e9b0 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 10 client.25517 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 10 client.25517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 20 client.25517 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b969fd0 0x56539b96c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 4 client.25517 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.385+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56539b8c88b0 con 0x56539b969fd0 2024-04-25T18:49:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.386+0000 7f95a6000640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b969fd0 0x56539b96c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.386+0000 7f95a6000640 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b969fd0 0x56539b96c490 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20983961d15e1f6a server_cookie=ae8de2930ff777bb in_seq=0 out_seq=0 2024-04-25T18:49:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.386+0000 7f959cc00640 10 client.25517 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.387+0000 7f959cc00640 1 -- 172.21.15.63:0/3302618859 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56539b8c88b0 con 0x56539b969fd0 2024-04-25T18:49:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.387+0000 7f959cc00640 10 client.25517 handle_command_reply: tid=0 2024-04-25T18:49:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 client.25517 shutdown 2024-04-25T18:49:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 msgr2=0x56539b956b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b956b30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f95a00ce570 tx=0x7f95a00d1030 comp rx=0 tx=0).stop 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 shutdown_connections 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b969fd0 0x56539b96c490 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b959b90 0x56539b9575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b957070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 --2- 172.21.15.63:0/3302618859 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b94e9b0 0x56539b956b30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f959cc00640 0 client.25517 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 >> 172.21.15.63:0/3302618859 conn(0x56539b94aec0 msgr2=0x56539b950c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 shutdown_connections 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 1 -- 172.21.15.63:0/3302618859 wait complete. 2024-04-25T18:49:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.388+0000 7f95b860d740 20 client.25517 trim_cache size 0 max 16384 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 Processor -- start 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 -- start start 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b956410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01b2580 con 0x56539b94f2d0 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01afe50 con 0x56539b959b90 2024-04-25T18:49:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95b860d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01adca0 con 0x56539b94e9b0 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95a6000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b956410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95aea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56104/0 (socket says 172.21.15.63:56104) 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.406+0000 7f95a6000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b956410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34916/0 (socket says 172.21.15.63:34916) 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 -- 172.21.15.63:0/1568585353 learned_addr learned my addr 172.21.15.63:0/1568585353 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95af400640 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 -- 172.21.15.63:0/1568585353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 msgr2=0x56539b9513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 -- 172.21.15.63:0/1568585353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 msgr2=0x56539b956410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b956410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 -- 172.21.15.63:0/1568585353 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56539b95bf80 con 0x56539b94e9b0 2024-04-25T18:49:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95af400640 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.407+0000 7f95aea00640 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f95a0099220 tx=0x7f95a009ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80e961fd309d19e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.408+0000 7f959f400640 1 -- 172.21.15.63:0/1568585353 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a009c900 con 0x56539b94e9b0 2024-04-25T18:49:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.408+0000 7f959f400640 1 -- 172.21.15.63:0/1568585353 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a009d4c0 con 0x56539b94e9b0 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.408+0000 7f959f400640 1 -- 172.21.15.63:0/1568585353 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a00a55a0 con 0x56539b94e9b0 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.408+0000 7f95b860d740 1 -- 172.21.15.63:0/1568585353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 msgr2=0x56539b958e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.408+0000 7f95b860d740 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f95a0099220 tx=0x7f95a009ccc0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 -- 172.21.15.63:0/1568585353 shutdown_connections 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94f2d0 0x56539b956410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 --2- 172.21.15.63:0/1568585353 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94e9b0 0x56539b958e30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 -- 172.21.15.63:0/1568585353 >> 172.21.15.63:0/1568585353 conn(0x56539b94aec0 msgr2=0x56539b957380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 -- 172.21.15.63:0/1568585353 shutdown_connections 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 -- 172.21.15.63:0/1568585353 wait complete. 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 -- start start 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.409+0000 7f95b860d740 1 Processor -- start 2024-04-25T18:49:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 0x56539b95ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01ae310 con 0x56539b94e9b0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01af780 con 0x56539b959b90 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f95b01ae9e0 con 0x56539b94f2d0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95b860d740 10 client.0 _ll_register_callbacks cb 0x7f95a00d4670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95a6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95af400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95a6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56106/0 (socket says 172.21.15.63:56106) 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95a6000640 1 -- 172.21.15.63:0/3671971770 learned_addr learned my addr 172.21.15.63:0/3671971770 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.410+0000 7f95aea00640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 0x56539b95ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 -- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 msgr2=0x56539b9ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 -- 172.21.15.63:0/3671971770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 msgr2=0x56539b95ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 0x56539b95ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 -- 172.21.15.63:0/3671971770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56539b9ceb40 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95aea00640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 0x56539b95ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95a6000640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91eb95743b963266 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f959cc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980050d0 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95a009d4c0 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.411+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f9598005270 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9598005470 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 10 client.25546 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56539b9cfe30 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95a009d4c0 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f95980035f0 con 0x56539b94f2d0 2024-04-25T18:49:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 10 client.25546 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 10 client.25546 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 20 client.25546 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b9d0260 0x56539b9d2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 4 client.25546 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56539b8c2320 con 0x56539b9d0260 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.412+0000 7f95aea00640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b9d0260 0x56539b9d2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.413+0000 7f95aea00640 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b9d0260 0x56539b9d2680 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=336fedaaa4f9aad8 server_cookie=e8586a7a680f3760 in_seq=0 out_seq=0 2024-04-25T18:49:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.413+0000 7f959cc00640 10 client.25546 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 26053 2.62 MB/sec warmup 60 sec latency 31232.350 ms 2024-04-25T18:49:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.438+0000 7f959cc00640 1 -- 172.21.15.63:0/3671971770 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56539b8c2320 con 0x56539b9d0260 2024-04-25T18:49:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.438+0000 7f959cc00640 10 client.25546 handle_command_reply: tid=0 2024-04-25T18:49:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.438+0000 7f95b860d740 1 client.25546 shutdown 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 msgr2=0x56539b95f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598009530 comp rx=0 tx=0).stop 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 shutdown_connections 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56539b9d0260 0x56539b9d2680 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56539b959b90 0x56539b9ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56539b94f2d0 0x56539b95f290 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 --2- 172.21.15.63:0/3671971770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56539b94e9b0 0x56539b95ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f959cc00640 0 client.25546 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 >> 172.21.15.63:0/3671971770 conn(0x56539b94aec0 msgr2=0x56539b949790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 shutdown_connections 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 1 -- 172.21.15.63:0/3671971770 wait complete. 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.439+0000 7f95b860d740 20 client.25546 trim_cache size 0 max 16384 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:49:01.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 msgr2=0x7f95940a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 0x7f95940a6250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f95a802a9a0 tx=0x7f95a8005a40 comp rx=0 tx=0).stop 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 msgr2=0x7f95b01a0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f95a000eb90 tx=0x7f95a000c6a0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 shutdown_connections 2024-04-25T18:49:01.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f95940a3d90 0x7f95940a6250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f95b010e610 0x7f95b019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95b0074c40 0x7f95b01a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 --2- 172.21.15.63:0/39404258 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f95b0074320 0x7f95b01a0430 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 >> 172.21.15.63:0/39404258 conn(0x7f95b00fd8a0 msgr2=0x7f95b00fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 shutdown_connections 2024-04-25T18:49:01.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:01.440+0000 7f95afe00640 1 -- 172.21.15.63:0/39404258 wait complete. 2024-04-25T18:49:02.175 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c672114c', seconds elapsed: 41.4 2024-04-25T18:49:02.175 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.15 2024-04-25T18:49:02.175 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:49:02.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:02 smithi063 ceph-mon[28006]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:02.436 INFO:tasks.workunit.client.0.smithi063.stdout: 1 27878 2.74 MB/sec warmup 61 sec latency 40.109 ms 2024-04-25T18:49:02.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:02 smithi114 ceph-mon[32988]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:02.659 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:49:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:02 smithi070 ceph-mon[32820]: pgmap v77: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:03.437 INFO:tasks.workunit.client.0.smithi063.stdout: 1 29016 2.81 MB/sec warmup 62 sec latency 16.794 ms 2024-04-25T18:49:04.437 INFO:tasks.workunit.client.0.smithi063.stdout: 1 30509 2.85 MB/sec warmup 63 sec latency 15.223 ms 2024-04-25T18:49:04.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:04 smithi114 ceph-mon[32988]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:04.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:04 smithi063 ceph-mon[28006]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:04 smithi070 ceph-mon[32820]: pgmap v78: 97 pgs: 97 active+clean; 27 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 -- 172.21.15.63:0/4155678361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0108f80 msgr2=0x7fe0e0109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 --2- 172.21.15.63:0/4155678361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0108f80 0x7fe0e0109400 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009990 tx=0x7fe0cc009ca0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 -- 172.21.15.63:0/4155678361 shutdown_connections 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 --2- 172.21.15.63:0/4155678361 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e010f030 0x7fe0e010f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 --2- 172.21.15.63:0/4155678361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e0109940 0x7fe0e0115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 --2- 172.21.15.63:0/4155678361 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0108f80 0x7fe0e0109400 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 -- 172.21.15.63:0/4155678361 >> 172.21.15.63:0/4155678361 conn(0x7fe0e00787d0 msgr2=0x7fe0e0107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:05.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 -- 172.21.15.63:0/4155678361 shutdown_connections 2024-04-25T18:49:05.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.226+0000 7fe0e6600700 1 -- 172.21.15.63:0/4155678361 wait complete. 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 Processor -- start 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 -- start start 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 0x7fe0e019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 0x7fe0e01a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e0119140 con 0x7fe0e010f030 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e0118fc0 con 0x7fe0e0109940 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.228+0000 7fe0e6600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e01192c0 con 0x7fe0e0108f80 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0e5200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 0x7fe0e019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57926/0 (socket says 172.21.15.63:57926) 2024-04-25T18:49:05.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 -- 172.21.15.63:0/1316612249 learned_addr learned my addr 172.21.15.63:0/1316612249 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:05.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0e5c00700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 0x7fe0e01a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 -- 172.21.15.63:0/1316612249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 msgr2=0x7fe0e019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 0x7fe0e019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 -- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 msgr2=0x7fe0e01a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 0x7fe0e01a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 -- 172.21.15.63:0/1316612249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e01a45c0 con 0x7fe0e0109940 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0e5c00700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 0x7fe0e01a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0e5200700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 0x7fe0e019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.229+0000 7fe0dfe00700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005910 tx=0x7fe0cc00b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.230+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc027070 con 0x7fe0e0109940 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.230+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0cc022470 con 0x7fe0e0109940 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.230+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e01a4850 con 0x7fe0e0109940 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.230+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc017650 con 0x7fe0e0109940 2024-04-25T18:49:05.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.230+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e01ac130 con 0x7fe0e0109940 2024-04-25T18:49:05.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.232+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e0069730 con 0x7fe0e0109940 2024-04-25T18:49:05.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.233+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe0cc02a070 con 0x7fe0e0109940 2024-04-25T18:49:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.234+0000 7fe0dd600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 0x7fe0c80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.235+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fe0cc0d1510 con 0x7fe0e0109940 2024-04-25T18:49:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.235+0000 7fe0e5200700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 0x7fe0c80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:05.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.236+0000 7fe0e5200700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 0x7fe0c80a62f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4011490 tx=0x7fe0d4009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:05.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.238+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0cc013070 con 0x7fe0e0109940 2024-04-25T18:49:05.437 INFO:tasks.workunit.client.0.smithi063.stdout: 1 32191 3.01 MB/sec warmup 64 sec latency 11.930 ms 2024-04-25T18:49:05.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.534+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0e002df80 con 0x7fe0e0109940 2024-04-25T18:49:05.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.535+0000 7fe0dd600700 1 -- 172.21.15.63:0/1316612249 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11871 (secure 0 0 0) 0x7fe0cc09dbd0 con 0x7fe0e0109940 2024-04-25T18:49:05.538 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:49:05.538 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:48:08.989491+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 msgr2=0x7fe0c80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 0x7fe0c80a62f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4011490 tx=0x7fe0d4009260 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 msgr2=0x7fe0e019faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005910 tx=0x7fe0cc00b670 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 shutdown_connections 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe0c80a3e30 0x7fe0c80a62f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe0e0108f80 0x7fe0e019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e010f030 0x7fe0e01a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 --2- 172.21.15.63:0/1316612249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0e0109940 0x7fe0e019faf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 >> 172.21.15.63:0/1316612249 conn(0x7fe0e00787d0 msgr2=0x7fe0e0079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:05.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.538+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 shutdown_connections 2024-04-25T18:49:05.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:05.539+0000 7fe0e6600700 1 -- 172.21.15.63:0/1316612249 wait complete. 2024-04-25T18:49:05.542 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 25 2024-04-25T18:49:06.437 INFO:tasks.workunit.client.0.smithi063.stdout: 1 33148 3.00 MB/sec warmup 65 sec latency 15.589 ms 2024-04-25T18:49:06.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:06 smithi063 ceph-mon[28006]: pgmap v79: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 297 KiB/s wr, 68 op/s 2024-04-25T18:49:06.526 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:06 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1316612249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:06 smithi114 ceph-mon[32988]: pgmap v79: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 297 KiB/s wr, 68 op/s 2024-04-25T18:49:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:06 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1316612249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:06.612 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 83.15 --await-for 42.0' 2024-04-25T18:49:06.613 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 83.15 --await-for 42.0 2024-04-25T18:49:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:06 smithi070 ceph-mon[32820]: pgmap v79: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 297 KiB/s wr, 68 op/s 2024-04-25T18:49:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:06 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1316612249' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:07.099 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:49:07.437 INFO:tasks.workunit.client.0.smithi063.stdout: 1 34974 3.12 MB/sec warmup 66 sec latency 14.272 ms 2024-04-25T18:49:08.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 36169 3.17 MB/sec warmup 67 sec latency 16.039 ms 2024-04-25T18:49:08.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:08 smithi114 ceph-mon[32988]: pgmap v80: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 297 KiB/s wr, 66 op/s 2024-04-25T18:49:08.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:08 smithi063 ceph-mon[28006]: pgmap v80: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 297 KiB/s wr, 66 op/s 2024-04-25T18:49:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:08 smithi070 ceph-mon[32820]: pgmap v80: 97 pgs: 97 active+clean; 29 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 297 KiB/s wr, 66 op/s 2024-04-25T18:49:09.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 37862 3.22 MB/sec warmup 68 sec latency 17.854 ms 2024-04-25T18:49:09.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.584+0000 7fdccd000700 1 -- 172.21.15.63:0/931232795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806a740 msgr2=0x7fdcc806abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.584+0000 7fdccd000700 1 --2- 172.21.15.63:0/931232795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806a740 0x7fdcc806abc0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009990 tx=0x7fdcb0009ca0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 -- 172.21.15.63:0/931232795 shutdown_connections 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 --2- 172.21.15.63:0/931232795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806b100 0x7fdcc810f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 --2- 172.21.15.63:0/931232795 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc8108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 --2- 172.21.15.63:0/931232795 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806a740 0x7fdcc806abc0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 -- 172.21.15.63:0/931232795 >> 172.21.15.63:0/931232795 conn(0x7fdcc80788e0 msgr2=0x7fdcc8078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 -- 172.21.15.63:0/931232795 shutdown_connections 2024-04-25T18:49:09.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.585+0000 7fdccd000700 1 -- 172.21.15.63:0/931232795 wait complete. 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.586+0000 7fdccd000700 1 Processor -- start 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.586+0000 7fdccd000700 1 -- start start 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806b100 0x7fdcc819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc8112c40 con 0x7fdcc806a740 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc8112ac0 con 0x7fdcc8108a60 2024-04-25T18:49:09.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdccd000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc8112dc0 con 0x7fdcc806b100 2024-04-25T18:49:09.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdcc6800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39612/0 (socket says 172.21.15.63:39612) 2024-04-25T18:49:09.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdcc5e00700 1 -- 172.21.15.63:0/1129752112 learned_addr learned my addr 172.21.15.63:0/1129752112 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:09.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.587+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806b100 0x7fdcc819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 -- 172.21.15.63:0/1129752112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806b100 msgr2=0x7fdcc819b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806b100 0x7fdcc819b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 -- 172.21.15.63:0/1129752112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 msgr2=0x7fdcc819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 -- 172.21.15.63:0/1129752112 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc81a0310 con 0x7fdcc806a740 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc6800700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:09.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800ef10 tx=0x7fdcb800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcbf400700 1 -- 172.21.15.63:0/1129752112 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb800c9c0 con 0x7fdcc806a740 2024-04-25T18:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcbf400700 1 -- 172.21.15.63:0/1129752112 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcb8019070 con 0x7fdcc806a740 2024-04-25T18:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc81a05a0 con 0x7fdcc806a740 2024-04-25T18:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.588+0000 7fdcbf400700 1 -- 172.21.15.63:0/1129752112 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8014480 con 0x7fdcc806a740 2024-04-25T18:49:09.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.589+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc81a7e80 con 0x7fdcc806a740 2024-04-25T18:49:09.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.593+0000 7fdcbf400700 1 -- 172.21.15.63:0/1129752112 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdcb8012050 con 0x7fdcc806a740 2024-04-25T18:49:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.594+0000 7fdcbf400700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 0x7fdcac0a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 Processor -- start 2024-04-25T18:49:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcbf400700 1 -- 172.21.15.63:0/1129752112 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fdcb80d0750 con 0x7fdcc806a740 2024-04-25T18:49:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 0x7fdcac0a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 -- start start 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x5610054b8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5610053fe020 con 0x5610054b6180 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610053a48c0 con 0x5610054aa9b0 2024-04-25T18:49:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.595+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610053f9fe0 con 0x5610054ab2d0 2024-04-25T18:49:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49520/0 (socket says 172.21.15.63:49520) 2024-04-25T18:49:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 -- 172.21.15.63:0/4157277066 learned_addr learned my addr 172.21.15.63:0/4157277066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcbd400700 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc6800700 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x5610054b8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 0x7fdcac0a6310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdcc81038f0 tx=0x7fdcb0005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 -- 172.21.15.63:0/4157277066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 msgr2=0x5610054b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 -- 172.21.15.63:0/4157277066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 msgr2=0x5610054b8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x5610054b8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.596+0000 7fdcc5e00700 1 -- 172.21.15.63:0/4157277066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56100539a810 con 0x5610054aa9b0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.597+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8099050 tx=0x7fdcb8098a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c3f417648d51cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.597+0000 7fdcb7400700 1 -- 172.21.15.63:0/4157277066 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb809cd80 con 0x5610054aa9b0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.597+0000 7fdcb7400700 1 -- 172.21.15.63:0/4157277066 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcb809c680 con 0x5610054aa9b0 2024-04-25T18:49:09.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.597+0000 7fdcb7400700 1 -- 172.21.15.63:0/4157277066 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb809cb40 con 0x5610054aa9b0 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/4157277066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 msgr2=0x5610054aad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8099050 tx=0x7fdcb8098a50 comp rx=0 tx=0).stop 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/4157277066 shutdown_connections 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x5610054b8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054aad90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/4157277066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/4157277066 >> 172.21.15.63:0/4157277066 conn(0x5610053d9730 msgr2=0x5610054a6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/4157277066 shutdown_connections 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.598+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/4157277066 wait complete. 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.599+0000 7fdcd7a4bb80 1 -- start start 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.599+0000 7fdcd7a4bb80 1 Processor -- start 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054aa9b0 0x5610054b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054b6180 0x5610054b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5610054b8c40 con 0x5610054aa9b0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5610053b79f0 con 0x5610054b6180 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5610054b8dc0 con 0x5610054ab2d0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcd7a4bb80 10 client.0 _ll_register_callbacks cb 0x5610053968b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcc6800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054b6180 0x5610054b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49166/0 (socket says 172.21.15.63:49166) 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054aa9b0 0x5610054b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 -- 172.21.15.63:0/1855661758 learned_addr learned my addr 172.21.15.63:0/1855661758 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 -- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054b6180 msgr2=0x5610054b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054b6180 0x5610054b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 -- 172.21.15.63:0/1855661758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054aa9b0 msgr2=0x5610054b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054aa9b0 0x5610054b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.600+0000 7fdcbd400700 1 -- 172.21.15.63:0/1855661758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56100534ba60 con 0x5610054ab2d0 2024-04-25T18:49:09.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcbd400700 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0015000 tx=0x7fdcb0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=865d683aa35865c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcabe00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0003890 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fdcb0003a30 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcb809c680 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdcb0003c30 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcd7a4bb80 10 client.25564 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.601+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561005394c80 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcb809c680 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdcb00344c0 con 0x5610054ab2d0 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 10 client.25564 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 10 client.25564 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 20 client.25564 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:09.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610054c5930 0x5610054c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 4 client.25564 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:09.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56100540b810 con 0x5610054c5930 2024-04-25T18:49:09.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.602+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610054c5930 0x5610054c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.603+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610054c5930 0x5610054c7d50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c6fdb9815d0219 server_cookie=eb8ffe8346e05ea1 in_seq=0 out_seq=0 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.603+0000 7fdcabe00700 10 client.25564 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcabe00700 1 -- 172.21.15.63:0/1855661758 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56100540b810 con 0x5610054c5930 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcabe00700 10 client.25564 handle_command_reply: tid=0 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 client.25564 shutdown 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 msgr2=0x5610054b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0015000 tx=0x7fdcb0009310 comp rx=0 tx=0).stop 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 shutdown_connections 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610054c5930 0x5610054c7d50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054b6180 0x5610054b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054aa9b0 0x5610054b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/1855661758 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054b2bf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcabe00700 0 client.25564 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 >> 172.21.15.63:0/1855661758 conn(0x5610053d9730 msgr2=0x5610054a6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 shutdown_connections 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.604+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/1855661758 wait complete. 2024-04-25T18:49:09.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.605+0000 7fdcd7a4bb80 20 client.25564 trim_cache size 0 max 16384 2024-04-25T18:49:09.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.627+0000 7fdcd7a4bb80 1 Processor -- start 2024-04-25T18:49:09.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 -- start start 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054ab2d0 0x5610054b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 0x5610054cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81b1e80 con 0x5610054ab2d0 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81af710 con 0x5610054aa9b0 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81ad560 con 0x5610054b6180 2024-04-25T18:49:09.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49550/0 (socket says 172.21.15.63:49550) 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2061836136 learned_addr learned my addr 172.21.15.63:0/2061836136 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcbd400700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054ab2d0 0x5610054b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.628+0000 7fdcc6800700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 0x5610054cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2061836136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 msgr2=0x5610054cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 0x5610054cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2061836136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054ab2d0 msgr2=0x5610054b4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054ab2d0 0x5610054b4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2061836136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610054c83b0 con 0x5610054aa9b0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc6800700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 0x5610054cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcb809c380 tx=0x7fdcb8009eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b11ffe1c9255d38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcb6a00700 1 -- 172.21.15.63:0/2061836136 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80b9020 con 0x5610054aa9b0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcb6a00700 1 -- 172.21.15.63:0/2061836136 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcb809cd80 con 0x5610054aa9b0 2024-04-25T18:49:09.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.629+0000 7fdcb6a00700 1 -- 172.21.15.63:0/2061836136 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb809c8e0 con 0x5610054aa9b0 2024-04-25T18:49:09.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2061836136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 msgr2=0x5610054c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdcb809c380 tx=0x7fdcb8009eb0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2061836136 shutdown_connections 2024-04-25T18:49:09.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054b6180 0x5610054cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054c1b40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2061836136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054ab2d0 0x5610054b4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2061836136 >> 172.21.15.63:0/2061836136 conn(0x5610053d9730 msgr2=0x5610054b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2061836136 shutdown_connections 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.630+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2061836136 wait complete. 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 -- start start 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 Processor -- start 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054bb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x561005534690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81adbd0 con 0x5610054b6180 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81af040 con 0x5610054aa9b0 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.631+0000 7fdcd7a4bb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc81ae2a0 con 0x5610054ab2d0 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcd7a4bb80 10 client.0 _ll_register_callbacks cb 0x5610053968b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:09.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcbd400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054bb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc6800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x561005534690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49552/0 (socket says 172.21.15.63:49552) 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2976340843 learned_addr learned my addr 172.21.15.63:0/2976340843 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2976340843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 msgr2=0x5610054bb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054bb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2976340843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 msgr2=0x561005534690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x561005534690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.632+0000 7fdcc5e00700 1 -- 172.21.15.63:0/2976340843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561005534bd0 con 0x5610054aa9b0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcc5e00700 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8098ae0 tx=0x7fdcb809c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a38511fe5cb2fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80b9070 con 0x5610054aa9b0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7fdcb809a980 con 0x5610054aa9b0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcb809cd80 con 0x5610054aa9b0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdcb809c8e0 con 0x5610054aa9b0 2024-04-25T18:49:09.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 10 client.25547 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561005535e80 con 0x5610054aa9b0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcb809cd80 con 0x5610054aa9b0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdcb8085c50 con 0x5610054aa9b0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 10 client.25547 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 10 client.25547 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.633+0000 7fdcd7a4bb80 20 client.25547 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610055360b0 0x5610055384a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcd7a4bb80 4 client.25547 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.15, "await_for": 42.0}] 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 83.15, "await_for": 42.0}) -- 0x5610054058b0 con 0x5610055360b0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcbd400700 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610055360b0 0x5610055384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcbd400700 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610055360b0 0x5610055384a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6966b91e086d4f7 server_cookie=41a3fc90e3a63d32 in_seq=0 out_seq=0 2024-04-25T18:49:09.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.634+0000 7fdcabe00700 10 client.25547 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:09.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.712+0000 7fdcabe00700 1 -- 172.21.15.63:0/2976340843 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5610054058b0 con 0x5610055360b0 2024-04-25T18:49:09.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.712+0000 7fdcabe00700 10 client.25547 handle_command_reply: tid=0 2024-04-25T18:49:09.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.712+0000 7fdcd7a4bb80 1 client.25547 shutdown 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 msgr2=0x5610054bb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8098ae0 tx=0x7fdcb809c580 comp rx=0 tx=0).stop 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 shutdown_connections 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5610055360b0 0x5610055384a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5610054b6180 0x561005534690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5610054aa9b0 0x5610054bb200 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 --2- 172.21.15.63:0/2976340843 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5610054ab2d0 0x5610054bb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 >> 172.21.15.63:0/2976340843 conn(0x5610053d9730 msgr2=0x5610054b4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.713+0000 7fdcabe00700 0 client.25547 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.714+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 shutdown_connections 2024-04-25T18:49:09.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.714+0000 7fdcd7a4bb80 1 -- 172.21.15.63:0/2976340843 wait complete. 2024-04-25T18:49:09.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.714+0000 7fdcd7a4bb80 20 client.25547 trim_cache size 0 max 16384 2024-04-25T18:49:09.717 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 25, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 10, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "192487aa": { 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 10, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:49:09.718 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 83.2, 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:09.719 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 msgr2=0x7fdcac0a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 0x7fdcac0a6310 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdcc81038f0 tx=0x7fdcb0005fd0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 msgr2=0x7fdcc819b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800ef10 tx=0x7fdcb800c3a0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 shutdown_connections 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdcac0a3e50 0x7fdcac0a6310 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdcc8108a60 0x7fdcc819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcc806a740 0x7fdcc819b300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 --2- 172.21.15.63:0/1129752112 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdcc806b100 0x7fdcc819b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:09.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.715+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 >> 172.21.15.63:0/1129752112 conn(0x7fdcc80788e0 msgr2=0x7fdcc810dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.716+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 shutdown_connections 2024-04-25T18:49:09.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:09.716+0000 7fdccd000700 1 -- 172.21.15.63:0/1129752112 wait complete. 2024-04-25T18:49:10.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.23 MB/sec warmup 69 sec latency 774.256 ms 2024-04-25T18:49:10.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:10 smithi114 ceph-mon[32988]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 715 KiB/s wr, 135 op/s 2024-04-25T18:49:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:10 smithi063 ceph-mon[28006]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 715 KiB/s wr, 135 op/s 2024-04-25T18:49:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:10 smithi070 ceph-mon[32820]: pgmap v81: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 715 KiB/s wr, 135 op/s 2024-04-25T18:49:10.744 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '192487aa', quiesce took 8.6 seconds. Will release after: 23.15 2024-04-25T18:49:11.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.19 MB/sec warmup 70 sec latency 1774.359 ms 2024-04-25T18:49:12.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.14 MB/sec warmup 71 sec latency 2774.451 ms 2024-04-25T18:49:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:12 smithi114 ceph-mon[32988]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 133 op/s 2024-04-25T18:49:12.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:12 smithi063 ceph-mon[28006]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 133 op/s 2024-04-25T18:49:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:12 smithi070 ceph-mon[32820]: pgmap v82: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 133 op/s 2024-04-25T18:49:13.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.10 MB/sec warmup 72 sec latency 3774.594 ms 2024-04-25T18:49:14.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.06 MB/sec warmup 73 sec latency 4774.686 ms 2024-04-25T18:49:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:14 smithi114 ceph-mon[32988]: pgmap v83: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 132 op/s 2024-04-25T18:49:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 -- 172.21.15.63:0/1456071634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0101620 msgr2=0x7f65d0105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 --2- 172.21.15.63:0/1456071634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0101620 0x7f65d0105ae0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009860 tx=0x7f65b802a3d0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 -- 172.21.15.63:0/1456071634 shutdown_connections 2024-04-25T18:49:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 --2- 172.21.15.63:0/1456071634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0101620 0x7f65d0105ae0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.593 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 --2- 172.21.15.63:0/1456071634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0106100 0x7f65d0106590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.593 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 --2- 172.21.15.63:0/1456071634 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0100d00 0x7f65d01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.593 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 -- 172.21.15.63:0/1456071634 >> 172.21.15.63:0/1456071634 conn(0x7f65d00fc770 msgr2=0x7f65d00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.593 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 -- 172.21.15.63:0/1456071634 shutdown_connections 2024-04-25T18:49:14.593 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.590+0000 7f65d5200640 1 -- 172.21.15.63:0/1456071634 wait complete. 2024-04-25T18:49:14.595 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 Processor -- start 2024-04-25T18:49:14.595 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 -- start start 2024-04-25T18:49:14.595 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.595 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0101620 0x7f65d01a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.595 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0106100 0x7f65d019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65d0118500 con 0x7f65d0101620 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65d0118380 con 0x7f65d0100d00 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.591+0000 7f65d5200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65d0118680 con 0x7f65d0106100 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0101620 0x7f65d01a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49568/0 (socket says 172.21.15.63:49568) 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 -- 172.21.15.63:0/1994785994 learned_addr learned my addr 172.21.15.63:0/1994785994 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cfe00640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0106100 0x7f65d019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 -- 172.21.15.63:0/1994785994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0106100 msgr2=0x7f65d019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0106100 0x7f65d019e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 -- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0101620 msgr2=0x7f65d01a4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0101620 0x7f65d01a4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 -- 172.21.15.63:0/1994785994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d019f120 con 0x7f65d0100d00 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.592+0000 7f65cf400640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f65c000e9f0 tx=0x7f65c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.596 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.593+0000 7f65ccc00640 1 -- 172.21.15.63:0/1994785994 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c000cd80 con 0x7f65d0100d00 2024-04-25T18:49:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.593+0000 7f65ccc00640 1 -- 172.21.15.63:0/1994785994 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65c00108e0 con 0x7f65d0100d00 2024-04-25T18:49:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.593+0000 7f65ccc00640 1 -- 172.21.15.63:0/1994785994 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0018980 con 0x7f65d0100d00 2024-04-25T18:49:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.593+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d019f410 con 0x7f65d0100d00 2024-04-25T18:49:14.597 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.593+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d01ab6b0 con 0x7f65d0100d00 2024-04-25T18:49:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.597+0000 7f65d91fd740 1 Processor -- start 2024-04-25T18:49:14.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.597+0000 7f65d91fd740 1 -- start start 2024-04-25T18:49:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e86d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e872d0 0x55af97e91650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65ccc00640 1 -- 172.21.15.63:0/1994785994 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65c0010430 con 0x7f65d0100d00 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55af97dec900 con 0x55af97e869b0 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af97d92ef0 con 0x55af97e872d0 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65d91fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55af97de88c0 con 0x55af97e91b90 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65cfe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e86d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49214/0 (socket says 172.21.15.63:49214) 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65c6a00640 1 -- 172.21.15.63:0/1020779707 learned_addr learned my addr 172.21.15.63:0/1020779707 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.598+0000 7f65cf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e872d0 0x55af97e91650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 -- 172.21.15.63:0/1020779707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e872d0 msgr2=0x55af97e91650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e872d0 0x55af97e91650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 -- 172.21.15.63:0/1020779707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 msgr2=0x55af97e86d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.602 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e86d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.602 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 -- 172.21.15.63:0/1020779707 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af97aff210 con 0x55af97e91b90 2024-04-25T18:49:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c6a00640 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f65b802a8f0 tx=0x7f65b8030430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=493ab861b7548265 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c4c00640 1 -- 172.21.15.63:0/1020779707 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8030980 con 0x55af97e91b90 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c4c00640 1 -- 172.21.15.63:0/1020779707 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b80054f0 con 0x55af97e91b90 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.599+0000 7f65c4c00640 1 -- 172.21.15.63:0/1020779707 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8004b60 con 0x55af97e91b90 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65ccc00640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 0x7f65b40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65ccc00640 1 -- 172.21.15.63:0/1994785994 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f65c00d0b40 con 0x7f65d0100d00 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65cfe00640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 0x7f65b40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65d91fd740 1 -- 172.21.15.63:0/1020779707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 msgr2=0x55af97e93f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65d91fd740 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f65b802a8f0 tx=0x7f65b8030430 comp rx=0 tx=0).stop 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.600+0000 7f65d91fd740 1 -- 172.21.15.63:0/1020779707 shutdown_connections 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e872d0 0x55af97e91650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e91b90 0x55af97e93f80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 --2- 172.21.15.63:0/1020779707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e86d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 -- 172.21.15.63:0/1020779707 >> 172.21.15.63:0/1020779707 conn(0x55af97e82ec0 msgr2=0x55af97e832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 -- 172.21.15.63:0/1020779707 shutdown_connections 2024-04-25T18:49:14.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65cfe00640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 0x7f65b40a6250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65c8005fd0 tx=0x7f65c8010040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 -- 172.21.15.63:0/1020779707 wait complete. 2024-04-25T18:49:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 -- start start 2024-04-25T18:49:14.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.601+0000 7f65d91fd740 1 Processor -- start 2024-04-25T18:49:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 0x55af97e8efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97e8f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55af97e94760 con 0x55af97e872d0 2024-04-25T18:49:14.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55af97da6310 con 0x55af97e91b90 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55af97e948e0 con 0x55af97e869b0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65d91fd740 10 client.0 _ll_register_callbacks cb 0x55af97d74cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 0x55af97e8efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65cf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49222/0 (socket says 172.21.15.63:49222) 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65cf400640 1 -- 172.21.15.63:0/2527848455 learned_addr learned my addr 172.21.15.63:0/2527848455 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.602+0000 7f65cfe00640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97e8f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 -- 172.21.15.63:0/2527848455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 msgr2=0x55af97e8ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 -- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 msgr2=0x55af97e8f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97e8f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 -- 172.21.15.63:0/2527848455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af97bcb620 con 0x55af97e872d0 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65cf400640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65cfe00640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97e8f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 0x55af97e8efb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f65b8005e00 tx=0x7f65b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b428cfb2f623349 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65bd600640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8054060 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f65b80048e0 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.603+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65b80054f0 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f65b8030d90 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 10 client.15393 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55af97e94760 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b80054f0 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f65b8044020 con 0x55af97e872d0 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 10 client.15393 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 10 client.15393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 20 client.15393 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97ea1ac0 0x55af97ea3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 4 client.15393 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.604+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af97e008b0 con 0x55af97ea1ac0 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.605+0000 7f65cf400640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97ea1ac0 0x55af97ea3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.605+0000 7f65cf400640 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97ea1ac0 0x55af97ea3eb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de01c44493f57f29 server_cookie=634b32ec3166b in_seq=0 out_seq=0 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.605+0000 7f65bd600640 10 client.15393 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.606+0000 7f65bd600640 1 -- 172.21.15.63:0/2527848455 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55af97e008b0 con 0x55af97ea1ac0 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.606+0000 7f65bd600640 10 client.15393 handle_command_reply: tid=0 2024-04-25T18:49:14.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.606+0000 7f65d91fd740 1 client.15393 shutdown 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.606+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 msgr2=0x55af97e8efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.606+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 0x55af97e8efb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f65b8005e00 tx=0x7f65b80046c0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 shutdown_connections 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97ea1ac0 0x55af97ea3eb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97e8f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e872d0 0x55af97e8efb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2527848455 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e869b0 0x55af97e8ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65bd600640 0 client.15393 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 >> 172.21.15.63:0/2527848455 conn(0x55af97e82ec0 msgr2=0x55af97e88c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 shutdown_connections 2024-04-25T18:49:14.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 1 -- 172.21.15.63:0/2527848455 wait complete. 2024-04-25T18:49:14.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.607+0000 7f65d91fd740 20 client.15393 trim_cache size 0 max 16384 2024-04-25T18:49:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:14 smithi063 ceph-mon[28006]: pgmap v83: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 132 op/s 2024-04-25T18:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 Processor -- start 2024-04-25T18:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 -- start start 2024-04-25T18:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 0x55af97e9d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e91b90 0x55af97ea9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b6970 con 0x55af97e91b90 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b4240 con 0x55af97e869b0 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.626+0000 7f65d91fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b2090 con 0x55af97e872d0 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65cfe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e91b90 0x55af97ea9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65cf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 0x55af97e9d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:49224/0 (socket says 172.21.15.63:49224) 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 -- 172.21.15.63:0/2123503580 learned_addr learned my addr 172.21.15.63:0/2123503580 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 -- 172.21.15.63:0/2123503580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 msgr2=0x55af97e9d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 0x55af97e9d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 -- 172.21.15.63:0/2123503580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e91b90 msgr2=0x55af97ea9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e91b90 0x55af97ea9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 -- 172.21.15.63:0/2123503580 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af97e8cfc0 con 0x55af97e872d0 2024-04-25T18:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65cf400640 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 0x55af97e9d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:14.631 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.627+0000 7f65c6a00640 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f65b804f990 tx=0x7f65b8030830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f1e5806e5ed4be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.628+0000 7f65bfe00640 1 -- 172.21.15.63:0/2123503580 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8054070 con 0x55af97e872d0 2024-04-25T18:49:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.628+0000 7f65bfe00640 1 -- 172.21.15.63:0/2123503580 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b804fd80 con 0x55af97e872d0 2024-04-25T18:49:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.628+0000 7f65bfe00640 1 -- 172.21.15.63:0/2123503580 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8046440 con 0x55af97e872d0 2024-04-25T18:49:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.628+0000 7f65d91fd740 1 -- 172.21.15.63:0/2123503580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 msgr2=0x55af97ea95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.628+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f65b804f990 tx=0x7f65b8030830 comp rx=0 tx=0).stop 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 -- 172.21.15.63:0/2123503580 shutdown_connections 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e91b90 0x55af97ea9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97ea95a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 --2- 172.21.15.63:0/2123503580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e869b0 0x55af97e9d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 -- 172.21.15.63:0/2123503580 >> 172.21.15.63:0/2123503580 conn(0x55af97e82ec0 msgr2=0x55af97e89dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 -- 172.21.15.63:0/2123503580 shutdown_connections 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 -- 172.21.15.63:0/2123503580 wait complete. 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 -- start start 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.629+0000 7f65d91fd740 1 Processor -- start 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97e97330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97effe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b2700 con 0x55af97e869b0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b3b70 con 0x55af97e91b90 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65d01b2dd0 con 0x55af97e872d0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65d91fd740 10 client.0 _ll_register_callbacks cb 0x7f65c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65cf400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65c6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97e97330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65cf400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39690/0 (socket says 172.21.15.63:39690) 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65cf400640 1 -- 172.21.15.63:0/985628991 learned_addr learned my addr 172.21.15.63:0/985628991 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.630+0000 7f65cfe00640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97effe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 -- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 msgr2=0x55af97effe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97effe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 -- 172.21.15.63:0/985628991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 msgr2=0x55af97e96df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 -- 172.21.15.63:0/985628991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af97f00490 con 0x55af97e872d0 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65cf400640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65c6a00640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97e97330 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55af97e8d250 tx=0x7f65b804f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2deef2c30d4ca7c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65bd600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8054070 con 0x55af97e872d0 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f65b8005350 con 0x55af97e872d0 2024-04-25T18:49:14.636 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.631+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65b804fd80 con 0x55af97e872d0 2024-04-25T18:49:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f65b806e050 con 0x55af97e872d0 2024-04-25T18:49:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 10 client.25588 fetch_fsmap learned FSMap version 25 2024-04-25T18:49:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55af97f07850 con 0x55af97e872d0 2024-04-25T18:49:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65b804fd80 con 0x55af97e872d0 2024-04-25T18:49:14.637 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f65b8069030 con 0x55af97e872d0 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 10 client.25588 fetch_fsmap finished waiting for FSMap version 25 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 10 client.25588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 20 client.25588 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97f07da0 0x55af97f0a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 4 client.25588 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.632+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55af97dfa320 con 0x55af97f07da0 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.633+0000 7f65cf400640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97f07da0 0x55af97f0a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.633+0000 7f65cf400640 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97f07da0 0x55af97f0a1c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa2dde8eca295518 server_cookie=f2f9d32a13495382 in_seq=0 out_seq=0 2024-04-25T18:49:14.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.633+0000 7f65bd600640 10 client.25588 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65bd600640 1 -- 172.21.15.63:0/985628991 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55af97dfa320 con 0x55af97f07da0 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65bd600640 10 client.25588 handle_command_reply: tid=0 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 client.25588 shutdown 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 msgr2=0x55af97e97330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97e97330 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55af97e8d250 tx=0x7f65b804f9c0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 shutdown_connections 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55af97f07da0 0x55af97f0a1c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55af97e91b90 0x55af97effe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55af97e872d0 0x55af97e97330 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 --2- 172.21.15.63:0/985628991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55af97e869b0 0x55af97e96df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65bd600640 0 client.25588 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 >> 172.21.15.63:0/985628991 conn(0x55af97e82ec0 msgr2=0x55af97e81790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 shutdown_connections 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.675+0000 7f65d91fd740 1 -- 172.21.15.63:0/985628991 wait complete. 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.676+0000 7f65d91fd740 20 client.25588 trim_cache size 0 max 16384 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:49:14.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:49:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.676+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 msgr2=0x7f65b40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.676+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 0x7f65b40a6250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65c8005fd0 tx=0x7f65c8010040 comp rx=0 tx=0).stop 2024-04-25T18:49:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.676+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 msgr2=0x7f65d01a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.676+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f65c000e9f0 tx=0x7f65c000eeb0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 shutdown_connections 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65b40a3d90 0x7f65b40a6250 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65d0106100 0x7f65d019e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65d0101620 0x7f65d01a4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 --2- 172.21.15.63:0/1994785994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d0100d00 0x7f65d01a47f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 >> 172.21.15.63:0/1994785994 conn(0x7f65d00fc770 msgr2=0x7f65d00fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 shutdown_connections 2024-04-25T18:49:14.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:14.677+0000 7f65d5200640 1 -- 172.21.15.63:0/1994785994 wait complete. 2024-04-25T18:49:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:14 smithi070 ceph-mon[32820]: pgmap v83: 97 pgs: 97 active+clean; 33 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 713 KiB/s wr, 132 op/s 2024-04-25T18:49:15.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 3.02 MB/sec warmup 74 sec latency 5774.819 ms 2024-04-25T18:49:16.438 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.98 MB/sec warmup 75 sec latency 6774.922 ms 2024-04-25T18:49:16.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:16 smithi114 ceph-mon[32988]: pgmap v84: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 790 KiB/s wr, 153 op/s 2024-04-25T18:49:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:16 smithi063 ceph-mon[28006]: pgmap v84: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 790 KiB/s wr, 153 op/s 2024-04-25T18:49:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:16 smithi070 ceph-mon[32820]: pgmap v84: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 790 KiB/s wr, 153 op/s 2024-04-25T18:49:17.439 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.94 MB/sec warmup 76 sec latency 7775.069 ms 2024-04-25T18:49:18.439 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.90 MB/sec warmup 77 sec latency 8775.216 ms 2024-04-25T18:49:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:18 smithi114 ceph-mon[32988]: pgmap v85: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 494 KiB/s wr, 89 op/s 2024-04-25T18:49:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:18 smithi063 ceph-mon[28006]: pgmap v85: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 494 KiB/s wr, 89 op/s 2024-04-25T18:49:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:18 smithi070 ceph-mon[32820]: pgmap v85: 97 pgs: 97 active+clean; 34 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 494 KiB/s wr, 89 op/s 2024-04-25T18:49:19.439 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.86 MB/sec warmup 78 sec latency 9775.374 ms 2024-04-25T18:49:20.439 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.83 MB/sec warmup 79 sec latency 10775.480 ms 2024-04-25T18:49:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:20 smithi114 ceph-mon[32988]: pgmap v86: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 593 KiB/s wr, 111 op/s 2024-04-25T18:49:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:20 smithi063 ceph-mon[28006]: pgmap v86: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 593 KiB/s wr, 111 op/s 2024-04-25T18:49:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:20 smithi070 ceph-mon[32820]: pgmap v86: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 593 KiB/s wr, 111 op/s 2024-04-25T18:49:21.439 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.79 MB/sec warmup 80 sec latency 11775.601 ms 2024-04-25T18:49:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:22 smithi063 ceph-mon[28006]: pgmap v87: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:22 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:22.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.76 MB/sec warmup 81 sec latency 12776.142 ms 2024-04-25T18:49:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:22 smithi114 ceph-mon[32988]: pgmap v87: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:22 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:22 smithi070 ceph-mon[32820]: pgmap v87: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:22 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:23.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.72 MB/sec warmup 82 sec latency 13776.286 ms 2024-04-25T18:49:24.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.69 MB/sec warmup 83 sec latency 14776.406 ms 2024-04-25T18:49:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:24 smithi114 ceph-mon[32988]: pgmap v88: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:24 smithi063 ceph-mon[28006]: pgmap v88: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:24 smithi070 ceph-mon[32820]: pgmap v88: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 175 KiB/s wr, 42 op/s 2024-04-25T18:49:25.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.66 MB/sec warmup 84 sec latency 15776.547 ms 2024-04-25T18:49:26.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.63 MB/sec warmup 85 sec latency 16776.702 ms 2024-04-25T18:49:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:26 smithi114 ceph-mon[32988]: pgmap v89: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T18:49:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:26 smithi063 ceph-mon[28006]: pgmap v89: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T18:49:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:26 smithi070 ceph-mon[32820]: pgmap v89: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T18:49:27.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.60 MB/sec warmup 86 sec latency 17776.833 ms 2024-04-25T18:49:27.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 -- 172.21.15.63:0/1415090334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074ca0 msgr2=0x7f501410e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 --2- 172.21.15.63:0/1415090334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074ca0 0x7f501410e060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc009ca0 tx=0x7f4ffc02a950 comp rx=0 tx=0).stop 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 -- 172.21.15.63:0/1415090334 shutdown_connections 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 --2- 172.21.15.63:0/1415090334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074ca0 0x7f501410e060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 --2- 172.21.15.63:0/1415090334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 0x7f50141109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 --2- 172.21.15.63:0/1415090334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074380 0x7f5014074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 -- 172.21.15.63:0/1415090334 >> 172.21.15.63:0/1415090334 conn(0x7f50140fd890 msgr2=0x7f50140ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 -- 172.21.15.63:0/1415090334 shutdown_connections 2024-04-25T18:49:27.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.826+0000 7f5013200640 1 -- 172.21.15.63:0/1415090334 wait complete. 2024-04-25T18:49:27.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.827+0000 7f5013200640 1 Processor -- start 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.827+0000 7f5013200640 1 -- start start 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074ca0 0x7f50141a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 0x7f501419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5014114230 con 0x7f501410e5a0 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50141140b0 con 0x7f5014074380 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5013200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50141143b0 con 0x7f5014074ca0 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56236/0 (socket says 172.21.15.63:56236) 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 0x7f501419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 -- 172.21.15.63:0/16900168 learned_addr learned my addr 172.21.15.63:0/16900168 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011400640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074ca0 0x7f50141a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 -- 172.21.15.63:0/16900168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074ca0 msgr2=0x7f50141a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074ca0 0x7f50141a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 -- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 msgr2=0x7f501419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 0x7f501419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.828+0000 7f5011e00640 1 -- 172.21.15.63:0/16900168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501419ad50 con 0x7f5014074380 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f5011e00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500400e9f0 tx=0x7f500400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f500aa00640 1 -- 172.21.15.63:0/16900168 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500400cd80 con 0x7f5014074380 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f500aa00640 1 -- 172.21.15.63:0/16900168 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50040054d0 con 0x7f5014074380 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f500aa00640 1 -- 172.21.15.63:0/16900168 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004010690 con 0x7f5014074380 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501419b040 con 0x7f5014074380 2024-04-25T18:49:27.833 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.829+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141a72e0 con 0x7f5014074380 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.832+0000 7f501bbba740 1 Processor -- start 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.832+0000 7f501bbba740 1 -- start start 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f500aa00640 1 -- 172.21.15.63:0/16900168 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5004004e90 con 0x7f5014074380 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58908650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564c58863900 con 0x564c588fd9b0 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564c58809ef0 con 0x564c588fe2d0 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f501bbba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c5885f8c0 con 0x564c58908b90 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38304/0 (socket says 172.21.15.63:38304) 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f5012800640 1 -- 172.21.15.63:0/1980115052 learned_addr learned my addr 172.21.15.63:0/1980115052 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f5011400640 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.833+0000 7f5011e00640 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58908650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 -- 172.21.15.63:0/1980115052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 msgr2=0x564c5890af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 -- 172.21.15.63:0/1980115052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 msgr2=0x564c58908650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58908650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 -- 172.21.15.63:0/1980115052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c58576210 con 0x564c588fd9b0 2024-04-25T18:49:27.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5012800640 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f500c009c20 tx=0x7f500c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9a711576850d82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5003400640 1 -- 172.21.15.63:0/1980115052 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c01e070 con 0x564c588fd9b0 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5003400640 1 -- 172.21.15.63:0/1980115052 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f500c019ae0 con 0x564c588fd9b0 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f500aa00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 0x7f4ff80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.834+0000 7f5003400640 1 -- 172.21.15.63:0/1980115052 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c00e900 con 0x564c588fd9b0 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f500aa00640 1 -- 172.21.15.63:0/16900168 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f50040d0c90 con 0x7f5014074380 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f5011e00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 0x7f4ff80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 -- 172.21.15.63:0/1980115052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 msgr2=0x564c588fdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f500c009c20 tx=0x7f500c011810 comp rx=0 tx=0).stop 2024-04-25T18:49:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 -- 172.21.15.63:0/1980115052 shutdown_connections 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58908650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 --2- 172.21.15.63:0/1980115052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c588fdd90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 -- 172.21.15.63:0/1980115052 >> 172.21.15.63:0/1980115052 conn(0x564c588f9ec0 msgr2=0x564c588fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 -- 172.21.15.63:0/1980115052 shutdown_connections 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.835+0000 7f501bbba740 1 -- 172.21.15.63:0/1980115052 wait complete. 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.836+0000 7f5011e00640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 0x7f4ff80a6250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x564c588fef50 tx=0x7f5004004a90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.836+0000 7f501bbba740 1 -- start start 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.836+0000 7f501bbba740 1 Processor -- start 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58905fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58906520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564c5890b7c0 con 0x564c58908b90 2024-04-25T18:49:27.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564c5881d310 con 0x564c588fe2d0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564c5890b940 con 0x564c588fd9b0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f501bbba740 10 client.0 _ll_register_callbacks cb 0x564c587ebcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58906520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5011400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58905fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:36568/0 (socket says 172.21.15.63:36568) 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 -- 172.21.15.63:0/630590863 learned_addr learned my addr 172.21.15.63:0/630590863 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 -- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 msgr2=0x564c58905fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58905fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 -- 172.21.15.63:0/630590863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 msgr2=0x564c58906520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58906520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.837+0000 7f5012800640 1 -- 172.21.15.63:0/630590863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c58642620 con 0x564c588fd9b0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f5012800640 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f500c011d10 tx=0x7f500c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236ca7fad760b47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f4ff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:27.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c02d060 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f500c019ae0 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f500c004230 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f500c01e040 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f501bbba740 10 client.25600 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.838+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564c589179e0 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f500c019ae0 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f500c004990 con 0x564c588fd9b0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 10 client.25600 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 10 client.25600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 20 client.25600 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c58918e50 0x564c5891b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 4 client.25600 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c588778b0 con 0x564c58918e50 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.839+0000 7f5011400640 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c58918e50 0x564c5891b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.840+0000 7f5011400640 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c58918e50 0x564c5891b240 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3ee75c409341c69 server_cookie=445ecac9c52d0779 in_seq=0 out_seq=0 2024-04-25T18:49:27.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.840+0000 7f4ff7e00640 10 client.25600 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f4ff7e00640 1 -- 172.21.15.63:0/630590863 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c588778b0 con 0x564c58918e50 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f4ff7e00640 10 client.25600 handle_command_reply: tid=0 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 client.25600 shutdown 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 msgr2=0x564c58905aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f500c011d10 tx=0x7f500c000bc0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 shutdown_connections 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c58918e50 0x564c5891b240 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58905fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58906520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 --2- 172.21.15.63:0/630590863 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58905aa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f4ff7e00640 0 client.25600 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 >> 172.21.15.63:0/630590863 conn(0x564c588f9ec0 msgr2=0x564c588ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 shutdown_connections 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 1 -- 172.21.15.63:0/630590863 wait complete. 2024-04-25T18:49:27.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.841+0000 7f501bbba740 20 client.25600 trim_cache size 0 max 16384 2024-04-25T18:49:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 Processor -- start 2024-04-25T18:49:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 -- start start 2024-04-25T18:49:27.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58914680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50141b2650 con 0x564c58908b90 2024-04-25T18:49:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50141afef0 con 0x564c588fe2d0 2024-04-25T18:49:27.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.858+0000 7f501bbba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50141add40 con 0x564c588fd9b0 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5011400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58914680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5011400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56254/0 (socket says 172.21.15.63:56254) 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38318/0 (socket says 172.21.15.63:38318) 2024-04-25T18:49:27.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5011400640 1 -- 172.21.15.63:0/3075910509 learned_addr learned my addr 172.21.15.63:0/3075910509 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 -- 172.21.15.63:0/3075910509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 msgr2=0x564c58914680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58914680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 -- 172.21.15.63:0/3075910509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 msgr2=0x564c58920930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5012800640 1 -- 172.21.15.63:0/3075910509 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c588f5820 con 0x564c58908b90 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.859+0000 7f5011400640 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.860+0000 7f5012800640 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f500c0059f0 tx=0x7f500c005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c85846291574c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.860+0000 7f5002a00640 1 -- 172.21.15.63:0/3075910509 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c02d070 con 0x564c58908b90 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.860+0000 7f5002a00640 1 -- 172.21.15.63:0/3075910509 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f500c00ee70 con 0x564c58908b90 2024-04-25T18:49:27.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.860+0000 7f5002a00640 1 -- 172.21.15.63:0/3075910509 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c005040 con 0x564c58908b90 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- 172.21.15.63:0/3075910509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 msgr2=0x564c58920e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f500c0059f0 tx=0x7f500c005b90 comp rx=0 tx=0).stop 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- 172.21.15.63:0/3075910509 shutdown_connections 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c58920930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c58908b90 0x564c58920e70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 --2- 172.21.15.63:0/3075910509 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c588fd9b0 0x564c58914680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- 172.21.15.63:0/3075910509 >> 172.21.15.63:0/3075910509 conn(0x564c588f9ec0 msgr2=0x564c58904370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- 172.21.15.63:0/3075910509 shutdown_connections 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- 172.21.15.63:0/3075910509 wait complete. 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.861+0000 7f501bbba740 1 -- start start 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 Processor -- start 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c5890e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50141ae3b0 con 0x564c588fd9b0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f50141af820 con 0x564c588fe2d0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f50141aea80 con 0x564c58908b90 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f501bbba740 10 client.0 _ll_register_callbacks cb 0x7f500c019760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f5012800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38326/0 (socket says 172.21.15.63:38326) 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.862+0000 7f5012800640 1 -- 172.21.15.63:0/372829719 learned_addr learned my addr 172.21.15.63:0/372829719 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5011400640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c5890e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 -- 172.21.15.63:0/372829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 msgr2=0x564c5890e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 -- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 msgr2=0x564c5890e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c5890e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 -- 172.21.15.63:0/372829719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c5890ebe0 con 0x564c588fd9b0 2024-04-25T18:49:27.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f5012800640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f500c004970 tx=0x7f500c004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9fe4b22240359c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f4ff7e00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c02d070 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f500c00ee70 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.863+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f500c00e430 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f500c019ab0 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 10 client.15417 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564c5897eeb0 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f500c00ee70 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f500c005500 con 0x564c588fd9b0 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 10 client.15417 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 10 client.15417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 20 client.15417 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c5897f540 0x564c58981960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 4 client.15417 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:49:27.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.864+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c58871320 con 0x564c5897f540 2024-04-25T18:49:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.865+0000 7f5011400640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c5897f540 0x564c58981960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.865+0000 7f5011400640 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c5897f540 0x564c58981960 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7b5bb4d59643aa8 server_cookie=9fbc546c8ba69de7 in_seq=0 out_seq=0 2024-04-25T18:49:27.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.865+0000 7f4ff7e00640 10 client.15417 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.899+0000 7f4ff7e00640 1 -- 172.21.15.63:0/372829719 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564c58871320 con 0x564c5897f540 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.899+0000 7f4ff7e00640 10 client.15417 handle_command_reply: tid=0 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.899+0000 7f501bbba740 1 client.15417 shutdown 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 msgr2=0x564c58910ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f500c004970 tx=0x7f500c004870 comp rx=0 tx=0).stop 2024-04-25T18:49:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 shutdown_connections 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564c5897f540 0x564c58981960 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564c588fe2d0 0x564c5890e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564c58908b90 0x564c5890e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 --2- 172.21.15.63:0/372829719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c588fd9b0 0x564c58910ca0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f4ff7e00640 0 client.15417 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 >> 172.21.15.63:0/372829719 conn(0x564c588f9ec0 msgr2=0x564c588f8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 shutdown_connections 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 1 -- 172.21.15.63:0/372829719 wait complete. 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.900+0000 7f501bbba740 20 client.15417 trim_cache size 0 max 16384 2024-04-25T18:49:27.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:49:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:49:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:49:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:49:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.901+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 msgr2=0x7f4ff80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.901+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 0x7f4ff80a6250 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x564c588fef50 tx=0x7f5004004a90 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.901+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 msgr2=0x7f50141a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.901+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f500400e9f0 tx=0x7f500400edf0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 shutdown_connections 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4ff80a3d90 0x7f4ff80a6250 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f501410e5a0 0x7f501419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5014074ca0 0x7f50141a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 --2- 172.21.15.63:0/16900168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5014074380 0x7f50141a0420 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 >> 172.21.15.63:0/16900168 conn(0x7f50140fd890 msgr2=0x7f50140ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 shutdown_connections 2024-04-25T18:49:27.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:27.902+0000 7f5013200640 1 -- 172.21.15.63:0/16900168 wait complete. 2024-04-25T18:49:28.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.57 MB/sec warmup 87 sec latency 18776.931 ms 2024-04-25T18:49:28.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:28 smithi114 ceph-mon[32988]: pgmap v90: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-04-25T18:49:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:28 smithi063 ceph-mon[28006]: pgmap v90: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-04-25T18:49:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:28 smithi070 ceph-mon[32820]: pgmap v90: 97 pgs: 97 active+clean; 31 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-04-25T18:49:29.440 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.54 MB/sec warmup 88 sec latency 19777.065 ms 2024-04-25T18:49:30.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.51 MB/sec warmup 89 sec latency 20777.212 ms 2024-04-25T18:49:30.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:30 smithi114 ceph-mon[32988]: pgmap v91: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 909 KiB/s wr, 55 op/s 2024-04-25T18:49:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:30 smithi063 ceph-mon[28006]: pgmap v91: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 909 KiB/s wr, 55 op/s 2024-04-25T18:49:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:30 smithi070 ceph-mon[32820]: pgmap v91: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 909 KiB/s wr, 55 op/s 2024-04-25T18:49:31.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.48 MB/sec warmup 90 sec latency 21777.376 ms 2024-04-25T18:49:32.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.45 MB/sec warmup 91 sec latency 22777.544 ms 2024-04-25T18:49:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:32 smithi114 ceph-mon[32988]: pgmap v92: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:32 smithi063 ceph-mon[28006]: pgmap v92: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:32 smithi070 ceph-mon[32820]: pgmap v92: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:33.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.43 MB/sec warmup 92 sec latency 23777.701 ms 2024-04-25T18:49:33.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:49:34.375 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:49:34.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.40 MB/sec warmup 93 sec latency 24777.794 ms 2024-04-25T18:49:34.563 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:34 smithi063 ceph-mon[28006]: pgmap v93: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:34.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:34 smithi114 ceph-mon[32988]: pgmap v93: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:34 smithi070 ceph-mon[32820]: pgmap v93: 97 pgs: 97 active+clean; 36 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 811 KiB/s wr, 33 op/s 2024-04-25T18:49:35.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.37 MB/sec warmup 94 sec latency 25777.904 ms 2024-04-25T18:49:36.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:36 smithi063 ceph-mon[28006]: pgmap v94: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 844 KiB/s wr, 47 op/s 2024-04-25T18:49:36.441 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.35 MB/sec warmup 95 sec latency 26778.003 ms 2024-04-25T18:49:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:36 smithi114 ceph-mon[32988]: pgmap v94: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 844 KiB/s wr, 47 op/s 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 -- 172.21.15.63:0/2477039244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f5806afe0 msgr2=0x7f8f5806b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 --2- 172.21.15.63:0/2477039244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f5806afe0 0x7f8f5806b440 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009990 tx=0x7f8f40009ca0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 -- 172.21.15.63:0/2477039244 shutdown_connections 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 --2- 172.21.15.63:0/2477039244 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f58108a20 0x7f8f58108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 --2- 172.21.15.63:0/2477039244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806b980 0x7f8f5810f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 --2- 172.21.15.63:0/2477039244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f5806afe0 0x7f8f5806b440 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 -- 172.21.15.63:0/2477039244 >> 172.21.15.63:0/2477039244 conn(0x7f8f580788b0 msgr2=0x7f8f58078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 -- 172.21.15.63:0/2477039244 shutdown_connections 2024-04-25T18:49:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.656+0000 7f8f5e200700 1 -- 172.21.15.63:0/2477039244 wait complete. 2024-04-25T18:49:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.658+0000 7f8f5e200700 1 Processor -- start 2024-04-25T18:49:36.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.658+0000 7f8f5e200700 1 -- start start 2024-04-25T18:49:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 0x7f8f5819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 0x7f8f5819b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f58108a20 0x7f8f5819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:36.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112bc0 con 0x7f8f58108a20 2024-04-25T18:49:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112a40 con 0x7f8f5806afe0 2024-04-25T18:49:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.659+0000 7f8f5e200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8f58112d40 con 0x7f8f5806b980 2024-04-25T18:49:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f4ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 0x7f8f5819b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:36.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f58108a20 0x7f8f5819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f4ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 0x7f8f5819b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:36594/0 (socket says 172.21.15.63:36594) 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 0x7f8f5819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f4ea00700 1 -- 172.21.15.63:0/3309596318 learned_addr learned my addr 172.21.15.63:0/3309596318 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 -- 172.21.15.63:0/3309596318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 msgr2=0x7f8f5819b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 0x7f8f5819b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 -- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f58108a20 msgr2=0x7f8f5819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f58108a20 0x7f8f5819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 -- 172.21.15.63:0/3309596318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f581a0330 con 0x7f8f5806afe0 2024-04-25T18:49:36.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.660+0000 7f8f57400700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 0x7f8f5819b370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800e9f0 tx=0x7f8f4800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.661+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800c9c0 con 0x7f8f5806afe0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.661+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f4800cb60 con 0x7f8f5806afe0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.661+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f48018680 con 0x7f8f5806afe0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.661+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f581a0620 con 0x7f8f5806afe0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.661+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f581a7ea0 con 0x7f8f5806afe0 2024-04-25T18:49:36.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.662+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f58069730 con 0x7f8f5806afe0 2024-04-25T18:49:36.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.665+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8f48010430 con 0x7f8f5806afe0 2024-04-25T18:49:36.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.666+0000 7f8f54c00700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 0x7f8f3c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:36.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.666+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f8f480d1110 con 0x7f8f5806afe0 2024-04-25T18:49:36.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.666+0000 7f8f4ea00700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 0x7f8f3c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:36.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.667+0000 7f8f4ea00700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 0x7f8f3c0a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f40005cd0 tx=0x7f8f40005c60 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:36.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.668+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f4809d0f0 con 0x7f8f5806afe0 2024-04-25T18:49:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:36 smithi070 ceph-mon[32820]: pgmap v94: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 844 KiB/s wr, 47 op/s 2024-04-25T18:49:36.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.954+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f5819c680 con 0x7f8f5806afe0 2024-04-25T18:49:36.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.955+0000 7f8f54c00700 1 -- 172.21.15.63:0/3309596318 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11872 (secure 0 0 0) 0x7f8f480a13f0 con 0x7f8f5806afe0 2024-04-25T18:49:36.956 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:49:36.956 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:49:21.158497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24496":{"gid":24496,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/3172988525","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":3172988525},{"type":"v1","addr":"172.21.15.114:6835","nonce":3172988525}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:49:36.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 msgr2=0x7f8f3c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:36.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 0x7f8f3c0a62f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8f40005cd0 tx=0x7f8f40005c60 comp rx=0 tx=0).stop 2024-04-25T18:49:36.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 msgr2=0x7f8f5819b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 0x7f8f5819b370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800e9f0 tx=0x7f8f4800ed00 comp rx=0 tx=0).stop 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 shutdown_connections 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8f3c0a3e30 0x7f8f3c0a62f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f58108a20 0x7f8f5819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f5806afe0 0x7f8f5819b370 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 --2- 172.21.15.63:0/3309596318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8f5806b980 0x7f8f5819b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 >> 172.21.15.63:0/3309596318 conn(0x7f8f580788b0 msgr2=0x7f8f5810df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 shutdown_connections 2024-04-25T18:49:36.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:36.958+0000 7f8f5e200700 1 -- 172.21.15.63:0/3309596318 wait complete. 2024-04-25T18:49:36.961 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 26 2024-04-25T18:49:37.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.32 MB/sec warmup 96 sec latency 27778.112 ms 2024-04-25T18:49:37.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:37 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3309596318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:37.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:37 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3309596318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:37 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3309596318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:49:37.968 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id 192487aa --release --await' 2024-04-25T18:49:37.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id 192487aa --release --await 2024-04-25T18:49:38.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.30 MB/sec warmup 97 sec latency 28778.654 ms 2024-04-25T18:49:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:49:38.598 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:38 smithi063 ceph-mon[28006]: pgmap v95: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 840 KiB/s wr, 44 op/s 2024-04-25T18:49:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:38 smithi070 ceph-mon[32820]: pgmap v95: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 840 KiB/s wr, 44 op/s 2024-04-25T18:49:38.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:38 smithi114 ceph-mon[32988]: pgmap v95: 97 pgs: 97 active+clean; 30 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 840 KiB/s wr, 44 op/s 2024-04-25T18:49:39.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.28 MB/sec warmup 98 sec latency 29778.718 ms 2024-04-25T18:49:40.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 38257 2.25 MB/sec warmup 99 sec latency 30778.814 ms 2024-04-25T18:49:40.559 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:40 smithi063 ceph-mon[28006]: pgmap v96: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 851 KiB/s wr, 48 op/s 2024-04-25T18:49:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:40 smithi070 ceph-mon[32820]: pgmap v96: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 851 KiB/s wr, 48 op/s 2024-04-25T18:49:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:40 smithi114 ceph-mon[32988]: pgmap v96: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 851 KiB/s wr, 48 op/s 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.861+0000 7f541b600700 1 -- 172.21.15.63:0/923234545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 msgr2=0x7f541410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.861+0000 7f541b600700 1 --2- 172.21.15.63:0/923234545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541410dcd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5400009990 tx=0x7f5400009ca0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 -- 172.21.15.63:0/923234545 shutdown_connections 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 --2- 172.21.15.63:0/923234545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 0x7f5414115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 --2- 172.21.15.63:0/923234545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f54141096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 --2- 172.21.15.63:0/923234545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541410dcd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 -- 172.21.15.63:0/923234545 >> 172.21.15.63:0/923234545 conn(0x7f5414076210 msgr2=0x7f5414107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 -- 172.21.15.63:0/923234545 shutdown_connections 2024-04-25T18:49:40.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.862+0000 7f541b600700 1 -- 172.21.15.63:0/923234545 wait complete. 2024-04-25T18:49:40.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.864+0000 7f541b600700 1 Processor -- start 2024-04-25T18:49:40.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.864+0000 7f541b600700 1 -- start start 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f541419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 0x7f54141a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5414119220 con 0x7f54141092d0 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54141190a0 con 0x7f5414109c80 2024-04-25T18:49:40.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f541b600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54141193a0 con 0x7f541410e3b0 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f5413e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f5418c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f541419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f5413e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40376/0 (socket says 172.21.15.63:40376) 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f5413e00700 1 -- 172.21.15.63:0/805380597 learned_addr learned my addr 172.21.15.63:0/805380597 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.865+0000 7f5419600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 0x7f54141a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 -- 172.21.15.63:0/805380597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 msgr2=0x7f54141a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 0x7f54141a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 -- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 msgr2=0x7f541419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f541419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 -- 172.21.15.63:0/805380597 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54141a4710 con 0x7f5414109c80 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5418c00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f541419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5413e00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54000098e0 tx=0x7f54000047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.866+0000 7f5411600700 1 -- 172.21.15.63:0/805380597 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400027070 con 0x7f5414109c80 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.867+0000 7f5411600700 1 -- 172.21.15.63:0/805380597 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5400022470 con 0x7f5414109c80 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.867+0000 7f5411600700 1 -- 172.21.15.63:0/805380597 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400017650 con 0x7f5414109c80 2024-04-25T18:49:40.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.867+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54141a49a0 con 0x7f5414109c80 2024-04-25T18:49:40.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.867+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54141ac280 con 0x7f5414109c80 2024-04-25T18:49:40.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.870+0000 7f5411600700 1 -- 172.21.15.63:0/805380597 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f540002a070 con 0x7f5414109c80 2024-04-25T18:49:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.871+0000 7f5411600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 0x7f53fc0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5411600700 1 -- 172.21.15.63:0/805380597 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f54000d1450 con 0x7f5414109c80 2024-04-25T18:49:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 Processor -- start 2024-04-25T18:49:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5418c00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 0x7f53fc0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 -- start start 2024-04-25T18:49:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1bd40 0x55943bb1c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb26f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55943ba6ef90 con 0x55943bb1c660 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55943ba15830 con 0x55943bb1bd40 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.872+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55943ba6af50 con 0x55943bb274d0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5418c00700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 0x7f53fc0a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5408011490 tx=0x7f5408009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5413e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb26f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41444/0 (socket says 172.21.15.63:41444) 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 -- 172.21.15.63:0/527889631 learned_addr learned my addr 172.21.15.63:0/527889631 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5418c00700 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1bd40 0x55943bb1c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 -- 172.21.15.63:0/527889631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1bd40 msgr2=0x55943bb1c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1bd40 0x55943bb1c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 -- 172.21.15.63:0/527889631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 msgr2=0x55943bb26f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb26f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5419600700 1 -- 172.21.15.63:0/527889631 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55943ba0b780 con 0x55943bb274d0 2024-04-25T18:49:40.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.873+0000 7f5413e00700 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb26f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:40.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.874+0000 7f5419600700 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f540c011630 tx=0x7f540c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=459db21cd1bb5fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.874+0000 7f5406000700 1 -- 172.21.15.63:0/527889631 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c011e70 con 0x55943bb274d0 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.874+0000 7f5406000700 1 -- 172.21.15.63:0/527889631 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f540c01a070 con 0x55943bb274d0 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.874+0000 7f5406000700 1 -- 172.21.15.63:0/527889631 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c007760 con 0x55943bb274d0 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 -- 172.21.15.63:0/527889631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 msgr2=0x55943bb298c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f540c011630 tx=0x7f540c009920 comp rx=0 tx=0).stop 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 -- 172.21.15.63:0/527889631 shutdown_connections 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb298c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1bd40 0x55943bb1c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/527889631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb26f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 -- 172.21.15.63:0/527889631 >> 172.21.15.63:0/527889631 conn(0x55943ba4b050 msgr2=0x55943bb18250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 -- 172.21.15.63:0/527889631 shutdown_connections 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.875+0000 7f5425fbbb80 1 -- 172.21.15.63:0/527889631 wait complete. 2024-04-25T18:49:40.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 -- start start 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 Processor -- start 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb26bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb24400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55943bb29f60 con 0x55943bb1bd40 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55943ba28960 con 0x55943bb1c660 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55943bb2a0e0 con 0x55943bb274d0 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.876+0000 7f5425fbbb80 10 client.0 _ll_register_callbacks cb 0x55943ba07820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:40.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40404/0 (socket says 172.21.15.63:40404) 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 -- 172.21.15.63:0/1339075319 learned_addr learned my addr 172.21.15.63:0/1339075319 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5419600700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb24400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5418c00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb26bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 -- 172.21.15.63:0/1339075319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 msgr2=0x55943bb24400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb24400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 -- 172.21.15.63:0/1339075319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 msgr2=0x55943bb26bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb26bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 -- 172.21.15.63:0/1339075319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55943b9bc9d0 con 0x55943bb1c660 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5418c00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb26bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f5413e00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5400099de0 tx=0x7f540009dc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=441596eaba78ebb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.877+0000 7f53faa00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000b5020 con 0x55943bb1c660 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f540009d670 con 0x55943bb1c660 2024-04-25T18:49:40.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f540c01a070 con 0x55943bb1c660 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f540009d870 con 0x55943bb1c660 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f5425fbbb80 10 client.25601 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55943ba05bf0 con 0x55943bb1c660 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.878+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f540c01a070 con 0x55943bb1c660 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f54000d1040 con 0x55943bb1c660 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 10 client.25601 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 10 client.25601 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 20 client.25601 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb36c20 0x55943bb39040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 4 client.25601 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:40.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55943ba7c780 con 0x55943bb36c20 2024-04-25T18:49:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.879+0000 7f5418c00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb36c20 0x55943bb39040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.880+0000 7f5418c00700 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb36c20 0x55943bb39040 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be535ab1aba0d68 server_cookie=1ea1e49632c57814 in_seq=0 out_seq=0 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.880+0000 7f53faa00700 10 client.25601 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f53faa00700 1 -- 172.21.15.63:0/1339075319 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55943ba7c780 con 0x55943bb36c20 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f53faa00700 10 client.25601 handle_command_reply: tid=0 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 client.25601 shutdown 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 msgr2=0x55943bb23ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5400099de0 tx=0x7f540009dc40 comp rx=0 tx=0).stop 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 shutdown_connections 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb36c20 0x55943bb39040 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb274d0 0x55943bb24400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb26bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f53faa00700 0 client.25601 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/1339075319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb1c660 0x55943bb23ec0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.881+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 >> 172.21.15.63:0/1339075319 conn(0x55943ba4b050 msgr2=0x55943bb17e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.882+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 shutdown_connections 2024-04-25T18:49:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.882+0000 7f5425fbbb80 1 -- 172.21.15.63:0/1339075319 wait complete. 2024-04-25T18:49:40.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.882+0000 7f5425fbbb80 20 client.25601 trim_cache size 0 max 16384 2024-04-25T18:49:40.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.904+0000 7f5425fbbb80 1 Processor -- start 2024-04-25T18:49:40.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.904+0000 7f5425fbbb80 1 -- start start 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.904+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 0x55943bb183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b6190 con 0x55943bb1c660 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b39f0 con 0x55943bb274d0 2024-04-25T18:49:40.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b1840 con 0x55943bb1bd40 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5419600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5419600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40420/0 (socket says 172.21.15.63:40420) 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45922/0 (socket says 172.21.15.63:45922) 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5419600700 1 -- 172.21.15.63:0/3413631277 learned_addr learned my addr 172.21.15.63:0/3413631277 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5418c00700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 0x55943bb183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 -- 172.21.15.63:0/3413631277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 msgr2=0x55943bb183a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 0x55943bb183a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 -- 172.21.15.63:0/3413631277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 msgr2=0x55943bb2f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5413e00700 1 -- 172.21.15.63:0/3413631277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55943bb1cce0 con 0x55943bb1c660 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.905+0000 7f5418c00700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 0x55943bb183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.906+0000 7f5419600700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.906+0000 7f5413e00700 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f540009d110 tx=0x7f5400022cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f280e4778e2efbed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.907+0000 7f5405600700 1 -- 172.21.15.63:0/3413631277 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000b5070 con 0x55943bb1c660 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.907+0000 7f5405600700 1 -- 172.21.15.63:0/3413631277 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5400099160 con 0x55943bb1c660 2024-04-25T18:49:40.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.907+0000 7f5405600700 1 -- 172.21.15.63:0/3413631277 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400022920 con 0x55943bb1c660 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3413631277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 msgr2=0x55943bb32e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f540009d110 tx=0x7f5400022cc0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3413631277 shutdown_connections 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1bd40 0x55943bb183a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3413631277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1c660 0x55943bb32e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3413631277 >> 172.21.15.63:0/3413631277 conn(0x55943ba4b050 msgr2=0x55943bb17c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3413631277 shutdown_connections 2024-04-25T18:49:40.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.908+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3413631277 wait complete. 2024-04-25T18:49:40.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 -- start start 2024-04-25T18:49:40.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 Processor -- start 2024-04-25T18:49:40.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 0x55943bb2aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b1eb0 con 0x55943bb1bd40 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b3320 con 0x55943bb274d0 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54141b2580 con 0x55943bb1c660 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.909+0000 7f5425fbbb80 10 client.0 _ll_register_callbacks cb 0x55943ba07820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:40.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5418c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5418c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45928/0 (socket says 172.21.15.63:45928) 2024-04-25T18:49:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5418c00700 1 -- 172.21.15.63:0/3486204479 learned_addr learned my addr 172.21.15.63:0/3486204479 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 0x55943bb2aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5419600700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 -- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 msgr2=0x55943bb2b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 -- 172.21.15.63:0/3486204479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 msgr2=0x55943bb2f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5413e00700 1 -- 172.21.15.63:0/3486204479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55943bb1dbb0 con 0x55943bb1c660 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5418c00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.910+0000 7f5419600700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f5413e00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 0x55943bb2aad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5400099870 tx=0x7f5400022b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5ca01124afa6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f53faa00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000b5070 con 0x55943bb1c660 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f540009ca70 con 0x55943bb1c660 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5400099160 con 0x55943bb1c660 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f54000996b0 con 0x55943bb1c660 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f5425fbbb80 10 client.25630 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:40.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55943bb1dbb0 con 0x55943bb1c660 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.911+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5400099160 con 0x55943bb1c660 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f54000d1040 con 0x55943bb1c660 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 10 client.25630 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 10 client.25630 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 20 client.25630 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb9f250 0x55943bba1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 4 client.25630 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "192487aa", "release": true, "await": true}] 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "192487aa", "release": true, "await": true}) -- 0x55943ba76820 con 0x55943bb9f250 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5418c00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb9f250 0x55943bba1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f5418c00700 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb9f250 0x55943bba1670 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ffce22fa6d85457 server_cookie=5be6983f9092275e in_seq=0 out_seq=0 2024-04-25T18:49:40.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.912+0000 7f53faa00700 10 client.25630 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f53faa00700 1 -- 172.21.15.63:0/3486204479 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55943ba76820 con 0x55943bb9f250 2024-04-25T18:49:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f53faa00700 10 client.25630 handle_command_reply: tid=0 2024-04-25T18:49:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f5425fbbb80 1 client.25630 shutdown 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 msgr2=0x55943bb2aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 0x55943bb2aad0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5400099870 tx=0x7f5400022b50 comp rx=0 tx=0).stop 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 shutdown_connections 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.918+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55943bb9f250 0x55943bba1670 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55943bb274d0 0x55943bb2b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f53faa00700 0 client.25630 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55943bb1bd40 0x55943bb2f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 --2- 172.21.15.63:0/3486204479 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55943bb1c660 0x55943bb2aad0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 >> 172.21.15.63:0/3486204479 conn(0x55943ba4b050 msgr2=0x55943bb31e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 shutdown_connections 2024-04-25T18:49:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 1 -- 172.21.15.63:0/3486204479 wait complete. 2024-04-25T18:49:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.919+0000 7f5425fbbb80 20 client.25630 trim_cache size 0 max 16384 2024-04-25T18:49:40.921 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:49:40.921 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 26, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 12, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "192487aa": { 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 12, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 83.2, 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:49:40.922 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:49:40.923 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.920+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 msgr2=0x7f53fc0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.920+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 0x7f53fc0a62f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5408011490 tx=0x7f5408009260 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.920+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 msgr2=0x7f541419fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.920+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f54000098e0 tx=0x7f54000047b0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 shutdown_connections 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f53fc0a3e30 0x7f53fc0a62f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f541410e3b0 0x7f54141a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54141092d0 0x7f541419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 --2- 172.21.15.63:0/805380597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5414109c80 0x7f541419fc90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 >> 172.21.15.63:0/805380597 conn(0x7f5414076210 msgr2=0x7f5414077dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 shutdown_connections 2024-04-25T18:49:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:49:40.921+0000 7f541b600700 1 -- 172.21.15.63:0/805380597 wait complete. 2024-04-25T18:49:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 -- 172.21.15.63:0/196053288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5081041f0 msgr2=0x7ff5081045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 --2- 172.21.15.63:0/196053288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5081041f0 0x7ff5081045d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8009900 tx=0x7ff4f802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 -- 172.21.15.63:0/196053288 shutdown_connections 2024-04-25T18:49:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 --2- 172.21.15.63:0/196053288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff508104ba0 0x7ff508108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 --2- 172.21.15.63:0/196053288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081092c0 0x7ff508110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 --2- 172.21.15.63:0/196053288 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff5081041f0 0x7ff5081045d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 -- 172.21.15.63:0/196053288 >> 172.21.15.63:0/196053288 conn(0x7ff5080ffa30 msgr2=0x7ff508101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 -- 172.21.15.63:0/196053288 shutdown_connections 2024-04-25T18:49:41.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.050+0000 7ff50ce00640 1 -- 172.21.15.63:0/196053288 wait complete. 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 Processor -- start 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 -- start start 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081041f0 0x7ff5081a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 0x7ff50819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff508114250 con 0x7ff5081041f0 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081140d0 con 0x7ff5081092c0 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.051+0000 7ff50ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081143d0 con 0x7ff50819a630 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff507400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081041f0 0x7ff5081a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff507e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 0x7ff50819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40450/0 (socket says 172.21.15.63:40450) 2024-04-25T18:49:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 -- 172.21.15.63:0/51038070 learned_addr learned my addr 172.21.15.63:0/51038070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 -- 172.21.15.63:0/51038070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 msgr2=0x7ff50819aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 0x7ff50819aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 -- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081041f0 msgr2=0x7ff5081a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081041f0 0x7ff5081a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff506a00640 1 -- 172.21.15.63:0/51038070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50819b150 con 0x7ff5081092c0 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.052+0000 7ff507e00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 0x7ff50819aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff506a00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e9a0 tx=0x7ff4f400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff4ffe00640 1 -- 172.21.15.63:0/51038070 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f400cd80 con 0x7ff5081092c0 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff4ffe00640 1 -- 172.21.15.63:0/51038070 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4f40035f0 con 0x7ff5081092c0 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff4ffe00640 1 -- 172.21.15.63:0/51038070 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4010690 con 0x7ff5081092c0 2024-04-25T18:49:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50819b440 con 0x7ff5081092c0 2024-04-25T18:49:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.053+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5081a75f0 con 0x7ff5081092c0 2024-04-25T18:49:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.056+0000 7ff511175740 1 Processor -- start 2024-04-25T18:49:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.056+0000 7ff511175740 1 -- start start 2024-04-25T18:49:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 0x55cf09411650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09413f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cf0936c900 con 0x55cf094072d0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf09312ef0 con 0x55cf094069b0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff511175740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf093688c0 con 0x55cf09411b90 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff4ffe00640 1 -- 172.21.15.63:0/51038070 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff4f4003aa0 con 0x7ff5081092c0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff507e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff507e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40460/0 (socket says 172.21.15.63:40460) 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff507e00640 1 -- 172.21.15.63:0/1465817228 learned_addr learned my addr 172.21.15.63:0/1465817228 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff507400640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 0x55cf09411650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.057+0000 7ff506a00640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09413f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 -- 172.21.15.63:0/1465817228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 msgr2=0x55cf09413f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09413f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 -- 172.21.15.63:0/1465817228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 msgr2=0x55cf09411650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 0x55cf09411650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 -- 172.21.15.63:0/1465817228 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf0907f210 con 0x55cf094069b0 2024-04-25T18:49:41.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff506a00640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09413f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:49:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507400640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 0x55cf09411650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff507e00640 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff500011790 tx=0x7ff500011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eacd7f220778e8d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff4fcc00640 1 -- 172.21.15.63:0/1465817228 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500004d20 con 0x55cf094069b0 2024-04-25T18:49:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff4fcc00640 1 -- 172.21.15.63:0/1465817228 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff500003970 con 0x55cf094069b0 2024-04-25T18:49:41.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.058+0000 7ff4fcc00640 1 -- 172.21.15.63:0/1465817228 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5000044c0 con 0x55cf094069b0 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff4ffe00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 0x7ff4ec0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff4ffe00640 1 -- 172.21.15.63:0/51038070 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff4f40d0ac0 con 0x7ff5081092c0 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 -- 172.21.15.63:0/1465817228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 msgr2=0x55cf09406d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff500011790 tx=0x7ff500011c50 comp rx=0 tx=0).stop 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff507400640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 0x7ff4ec0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 -- 172.21.15.63:0/1465817228 shutdown_connections 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09413f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094072d0 0x55cf09411650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 --2- 172.21.15.63:0/1465817228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094069b0 0x55cf09406d90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 -- 172.21.15.63:0/1465817228 >> 172.21.15.63:0/1465817228 conn(0x55cf09402ec0 msgr2=0x55cf094032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.059+0000 7ff511175740 1 -- 172.21.15.63:0/1465817228 shutdown_connections 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.060+0000 7ff511175740 1 -- 172.21.15.63:0/1465817228 wait complete. 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.060+0000 7ff511175740 1 -- start start 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.060+0000 7ff507400640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 0x7ff4ec0a6250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8005ab0 tx=0x7ff4f8005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.060+0000 7ff511175740 1 Processor -- start 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 0x55cf0940eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf0940f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf09411b90 0x55cf0940f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cf094147c0 con 0x55cf09411b90 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cf09326310 con 0x55cf094072d0 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cf09414940 con 0x55cf094069b0 2024-04-25T18:49:41.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff511175740 10 client.0 _ll_register_callbacks cb 0x55cf092f4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff507400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 0x55cf0940eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff506a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf0940f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff507400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 0x55cf0940eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41512/0 (socket says 172.21.15.63:41512) 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff507400640 1 -- 172.21.15.63:0/3334469224 learned_addr learned my addr 172.21.15.63:0/3334469224 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff507e00640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf09411b90 0x55cf0940f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff506a00640 1 -- 172.21.15.63:0/3334469224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 msgr2=0x55cf0940eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff506a00640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 0x55cf0940eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff506a00640 1 -- 172.21.15.63:0/3334469224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf09411b90 msgr2=0x55cf0940f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.061+0000 7ff506a00640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf09411b90 0x55cf0940f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff506a00640 1 -- 172.21.15.63:0/3334469224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf0914b620 con 0x55cf094072d0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff506a00640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf0940f060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40ce1a0 tx=0x7ff4f409c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce6ba0febe82294a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff4f1600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40ce620 con 0x55cf094072d0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff4f40a51d0 con 0x55cf094072d0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff500003970 con 0x55cf094072d0 2024-04-25T18:49:41.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff4f409c9f0 con 0x55cf094072d0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff511175740 10 client.25619 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cf09420990 con 0x55cf094072d0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.062+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff500003970 con 0x55cf094072d0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ff4f40870b0 con 0x55cf094072d0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 10 client.25619 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 10 client.25619 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 20 client.25619 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09421ea0 0x55cf094242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 4 client.25619 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf093808b0 con 0x55cf09421ea0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.063+0000 7ff507400640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09421ea0 0x55cf094242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.064+0000 7ff507400640 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09421ea0 0x55cf094242d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6b75f5806d05ce server_cookie=df88b26790d23fd8 in_seq=0 out_seq=0 2024-04-25T18:49:41.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.064+0000 7ff4f1600640 10 client.25619 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:41.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.064+0000 7ff4f1600640 1 -- 172.21.15.63:0/3334469224 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf093808b0 con 0x55cf09421ea0 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.064+0000 7ff4f1600640 10 client.25619 handle_command_reply: tid=0 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 client.25619 shutdown 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 msgr2=0x55cf0940f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf0940f060 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40ce1a0 tx=0x7ff4f409c350 comp rx=0 tx=0).stop 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 shutdown_connections 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09421ea0 0x55cf094242d0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf0940f060 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff4f1600640 0 client.25619 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf09411b90 0x55cf0940f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 --2- 172.21.15.63:0/3334469224 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094069b0 0x55cf0940eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 >> 172.21.15.63:0/3334469224 conn(0x55cf09402ec0 msgr2=0x55cf09408c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 shutdown_connections 2024-04-25T18:49:41.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 1 -- 172.21.15.63:0/3334469224 wait complete. 2024-04-25T18:49:41.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.065+0000 7ff511175740 20 client.25619 trim_cache size 0 max 16384 2024-04-25T18:49:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 Processor -- start 2024-04-25T18:49:41.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 -- start start 2024-04-25T18:49:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf0941d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf09411b90 0x55cf09410a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081b2690 con 0x55cf094069b0 2024-04-25T18:49:41.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081aff30 con 0x55cf09411b90 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.083+0000 7ff511175740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081add80 con 0x55cf094072d0 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff507400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf0941d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41522/0 (socket says 172.21.15.63:41522) 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 -- 172.21.15.63:0/2181548307 learned_addr learned my addr 172.21.15.63:0/2181548307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff507e00640 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf09411b90 0x55cf09410a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 -- 172.21.15.63:0/2181548307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf09411b90 msgr2=0x55cf09410a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf09411b90 0x55cf09410a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 -- 172.21.15.63:0/2181548307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 msgr2=0x55cf0941d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf0941d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.084+0000 7ff506a00640 1 -- 172.21.15.63:0/2181548307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf09424950 con 0x55cf094072d0 2024-04-25T18:49:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.085+0000 7ff506a00640 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400cfd0 tx=0x7ff4f409af00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c07a5ecc41e520e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.085+0000 7ff4f3e00640 1 -- 172.21.15.63:0/2181548307 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40b7030 con 0x55cf094072d0 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.085+0000 7ff4f3e00640 1 -- 172.21.15.63:0/2181548307 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4f40a5660 con 0x55cf094072d0 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.085+0000 7ff4f3e00640 1 -- 172.21.15.63:0/2181548307 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40bbbb0 con 0x55cf094072d0 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- 172.21.15.63:0/2181548307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 msgr2=0x55cf094093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400cfd0 tx=0x7ff4f409af00 comp rx=0 tx=0).stop 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- 172.21.15.63:0/2181548307 shutdown_connections 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf09411b90 0x55cf09410a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf094072d0 0x55cf094093c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 --2- 172.21.15.63:0/2181548307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf0941d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- 172.21.15.63:0/2181548307 >> 172.21.15.63:0/2181548307 conn(0x55cf09402ec0 msgr2=0x55cf09410350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- 172.21.15.63:0/2181548307 shutdown_connections 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- 172.21.15.63:0/2181548307 wait complete. 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.086+0000 7ff511175740 1 -- start start 2024-04-25T18:49:41.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 Processor -- start 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf094176c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf09417c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09418140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081ae3f0 con 0x55cf094069b0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081af860 con 0x55cf094072d0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff5081aeac0 con 0x55cf09411b90 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff511175740 10 client.0 _ll_register_callbacks cb 0x7ff4f409c7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff506a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf09417c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff507e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09418140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff507400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf094176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.091 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff506a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf09417c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40486/0 (socket says 172.21.15.63:40486) 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.087+0000 7ff506a00640 1 -- 172.21.15.63:0/1801118254 learned_addr learned my addr 172.21.15.63:0/1801118254 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 -- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 msgr2=0x55cf09417c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf09417c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 -- 172.21.15.63:0/1801118254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 msgr2=0x55cf094176c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf094176c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 -- 172.21.15.63:0/1801118254 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf09413f80 con 0x55cf09411b90 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507400640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf094176c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff507e00640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09418140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff50000f790 tx=0x7ff50000fc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dff9280ebd88b791 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff4f1600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50002a020 con 0x55cf09411b90 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4f40a5660 con 0x55cf09411b90 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.088+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7ff500003b70 con 0x55cf09411b90 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff500035050 con 0x55cf09411b90 2024-04-25T18:49:41.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 10 client.25645 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cf09488630 con 0x55cf09411b90 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4f40a5660 con 0x55cf09411b90 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7ff500023040 con 0x55cf09411b90 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 10 client.25645 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 10 client.25645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 20 client.25645 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09488ca0 0x55cf0948b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 4 client.25645 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf0937a320 con 0x55cf09488ca0 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.089+0000 7ff507400640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09488ca0 0x55cf0948b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.090+0000 7ff507400640 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09488ca0 0x55cf0948b0c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585e087b29870771 server_cookie=4ceaf7eef8cc4d95 in_seq=0 out_seq=0 2024-04-25T18:49:41.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.090+0000 7ff4f1600640 10 client.25645 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:41.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.108+0000 7ff4f1600640 1 -- 172.21.15.63:0/1801118254 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf0937a320 con 0x55cf09488ca0 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.108+0000 7ff4f1600640 10 client.25645 handle_command_reply: tid=0 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.108+0000 7ff511175740 1 client.25645 shutdown 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.108+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 msgr2=0x55cf09418140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.108+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09418140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff50000f790 tx=0x7ff50000fc90 comp rx=0 tx=0).stop 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 shutdown_connections 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cf09488ca0 0x55cf0948b0c0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cf094072d0 0x55cf09417c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff4f1600640 0 client.25645 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cf09411b90 0x55cf09418140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 --2- 172.21.15.63:0/1801118254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cf094069b0 0x55cf094176c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 >> 172.21.15.63:0/1801118254 conn(0x55cf09402ec0 msgr2=0x55cf09410be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 shutdown_connections 2024-04-25T18:49:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 1 -- 172.21.15.63:0/1801118254 wait complete. 2024-04-25T18:49:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.109+0000 7ff511175740 20 client.25645 trim_cache size 0 max 16384 2024-04-25T18:49:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:49:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 msgr2=0x7ff4ec0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 0x7ff4ec0a6250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8005ab0 tx=0x7ff4f8005a20 comp rx=0 tx=0).stop 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 msgr2=0x7ff5081a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:41.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400e9a0 tx=0x7ff4f400eda0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 shutdown_connections 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4ec0a3d90 0x7ff4ec0a6250 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff5081092c0 0x7ff5081a0970 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff50819a630 0x7ff50819aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 --2- 172.21.15.63:0/51038070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5081041f0 0x7ff5081a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 >> 172.21.15.63:0/51038070 conn(0x7ff5080ffa30 msgr2=0x7ff508100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 shutdown_connections 2024-04-25T18:49:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:41.110+0000 7ff50ce00640 1 -- 172.21.15.63:0/51038070 wait complete. 2024-04-25T18:49:41.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 39019 2.30 MB/sec warmup 100 sec latency 31261.314 ms 2024-04-25T18:49:42.091 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '192487aa', seconds elapsed: 39.9 2024-04-25T18:49:42.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 39882 2.30 MB/sec warmup 101 sec latency 17.810 ms 2024-04-25T18:49:42.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:42 smithi114 ceph-mon[32988]: pgmap v97: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:42 smithi063 ceph-mon[28006]: pgmap v97: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:42 smithi070 ceph-mon[32820]: pgmap v97: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:43.442 INFO:tasks.workunit.client.0.smithi063.stdout: 1 41565 2.34 MB/sec warmup 102 sec latency 13.784 ms 2024-04-25T18:49:44.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 42986 2.43 MB/sec warmup 103 sec latency 11.686 ms 2024-04-25T18:49:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:44 smithi063 ceph-mon[28006]: pgmap v98: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:44 smithi070 ceph-mon[32820]: pgmap v98: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:44.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:44 smithi114 ceph-mon[32988]: pgmap v98: 97 pgs: 97 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 17 op/s 2024-04-25T18:49:45.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 44178 2.46 MB/sec warmup 104 sec latency 21.900 ms 2024-04-25T18:49:46.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 46152 2.55 MB/sec warmup 105 sec latency 13.327 ms 2024-04-25T18:49:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:46 smithi063 ceph-mon[28006]: pgmap v99: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 286 KiB/s wr, 65 op/s 2024-04-25T18:49:46.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:46 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:46 smithi070 ceph-mon[32820]: pgmap v99: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 286 KiB/s wr, 65 op/s 2024-04-25T18:49:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:46 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:46 smithi114 ceph-mon[32988]: pgmap v99: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 286 KiB/s wr, 65 op/s 2024-04-25T18:49:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:46 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:49:47.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 47127 2.55 MB/sec warmup 106 sec latency 17.208 ms 2024-04-25T18:49:47.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:47 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:47.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:47 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:47.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:47 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:49:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:47 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:47 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:47 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:49:47.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:47 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:49:47.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:47 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:49:47.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:47 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:49:48.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 49009 2.62 MB/sec warmup 107 sec latency 14.596 ms 2024-04-25T18:49:48.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:48 smithi063 ceph-mon[28006]: pgmap v100: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 253 KiB/s wr, 52 op/s 2024-04-25T18:49:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:48 smithi070 ceph-mon[32820]: pgmap v100: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 253 KiB/s wr, 52 op/s 2024-04-25T18:49:48.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:48 smithi114 ceph-mon[32988]: pgmap v100: 97 pgs: 97 active+clean; 33 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 253 KiB/s wr, 52 op/s 2024-04-25T18:49:49.443 INFO:tasks.workunit.client.0.smithi063.stdout: 1 50159 2.67 MB/sec warmup 108 sec latency 18.916 ms 2024-04-25T18:49:50.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 51385 2.70 MB/sec warmup 109 sec latency 22.865 ms 2024-04-25T18:49:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:50 smithi063 ceph-mon[28006]: pgmap v101: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 KiB/s wr, 120 op/s 2024-04-25T18:49:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:50 smithi070 ceph-mon[32820]: pgmap v101: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 KiB/s wr, 120 op/s 2024-04-25T18:49:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:50 smithi114 ceph-mon[32988]: pgmap v101: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 666 KiB/s wr, 120 op/s 2024-04-25T18:49:51.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 53313 2.78 MB/sec warmup 110 sec latency 24.913 ms 2024-04-25T18:49:52.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 54216 2.78 MB/sec warmup 111 sec latency 44.921 ms 2024-04-25T18:49:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:52 smithi063 ceph-mon[28006]: pgmap v102: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:52 smithi070 ceph-mon[32820]: pgmap v102: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:52.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:52 smithi114 ceph-mon[32988]: pgmap v102: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:53.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 55839 2.82 MB/sec warmup 112 sec latency 18.873 ms 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 -- 172.21.15.63:0/4019235285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c107090 msgr2=0x7f7d3c10ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 --2- 172.21.15.63:0/4019235285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c107090 0x7f7d3c10ec60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009ca0 tx=0x7f7d3402a8c0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 -- 172.21.15.63:0/4019235285 shutdown_connections 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 --2- 172.21.15.63:0/4019235285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c107090 0x7f7d3c10ec60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 --2- 172.21.15.63:0/4019235285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c102970 0x7f7d3c1069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 --2- 172.21.15.63:0/4019235285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.266+0000 7f7d43000640 1 -- 172.21.15.63:0/4019235285 >> 172.21.15.63:0/4019235285 conn(0x7f7d3c0fd8a0 msgr2=0x7f7d3c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.267+0000 7f7d43000640 1 -- 172.21.15.63:0/4019235285 shutdown_connections 2024-04-25T18:49:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.267+0000 7f7d43000640 1 -- 172.21.15.63:0/4019235285 wait complete. 2024-04-25T18:49:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 Processor -- start 2024-04-25T18:49:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 -- start start 2024-04-25T18:49:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 0x7f7d3c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c19ce20 0x7f7d3c19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c112050 con 0x7f7d3c19ce20 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c111ed0 con 0x7f7d3c101fc0 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.268+0000 7f7d43000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1121d0 con 0x7f7d3c107090 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38972/0 (socket says 172.21.15.63:38972) 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 -- 172.21.15.63:0/2085829929 learned_addr learned my addr 172.21.15.63:0/2085829929 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41200640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 0x7f7d3c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d42600640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c19ce20 0x7f7d3c19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 -- 172.21.15.63:0/2085829929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 msgr2=0x7f7d3c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 0x7f7d3c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 -- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c19ce20 msgr2=0x7f7d3c19d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c19ce20 0x7f7d3c19d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 -- 172.21.15.63:0/2085829929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c19d910 con 0x7f7d3c101fc0 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41200640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 0x7f7d3c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:49:54.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d41c00640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e9f0 tx=0x7f7d2c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d3a000640 1 -- 172.21.15.63:0/2085829929 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c00cd80 con 0x7f7d3c101fc0 2024-04-25T18:49:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d3a000640 1 -- 172.21.15.63:0/2085829929 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2c0054d0 con 0x7f7d3c101fc0 2024-04-25T18:49:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.269+0000 7f7d3a000640 1 -- 172.21.15.63:0/2085829929 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c010690 con 0x7f7d3c101fc0 2024-04-25T18:49:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.270+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c19dc00 con 0x7f7d3c101fc0 2024-04-25T18:49:54.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.270+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c1a53a0 con 0x7f7d3c101fc0 2024-04-25T18:49:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.273+0000 7f7d3a000640 1 -- 172.21.15.63:0/2085829929 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7d2c004e90 con 0x7f7d3c101fc0 2024-04-25T18:49:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.273+0000 7f7d47586740 1 Processor -- start 2024-04-25T18:49:54.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.273+0000 7f7d47586740 1 -- start start 2024-04-25T18:49:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b60a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b63390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5575f0ab7900 con 0x5575f0b566e0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575f0a5def0 con 0x5575f0b60fa0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d47586740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5575f0ab38c0 con 0x5575f0b55dc0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d42600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d41200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b63390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d42600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44794/0 (socket says 172.21.15.63:44794) 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d42600640 1 -- 172.21.15.63:0/4281601120 learned_addr learned my addr 172.21.15.63:0/4281601120 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.274+0000 7f7d41c00640 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b60a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 -- 172.21.15.63:0/4281601120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 msgr2=0x5575f0b63390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b63390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 -- 172.21.15.63:0/4281601120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 msgr2=0x5575f0b60a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b60a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 -- 172.21.15.63:0/4281601120 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f07ca210 con 0x5575f0b55dc0 2024-04-25T18:49:54.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d41c00640 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b60a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d3a000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 0x7f7d200a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d42600640 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009ca0 tx=0x7f7d34009a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5fa8a19ef1a9515 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d3a000640 1 -- 172.21.15.63:0/2085829929 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7d2c0d0c70 con 0x7f7d3c101fc0 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d41c00640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 0x7f7d200a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d32a00640 1 -- 172.21.15.63:0/4281601120 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031cf0 con 0x5575f0b55dc0 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d32a00640 1 -- 172.21.15.63:0/4281601120 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d34036030 con 0x5575f0b55dc0 2024-04-25T18:49:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.275+0000 7f7d32a00640 1 -- 172.21.15.63:0/4281601120 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d340042a0 con 0x5575f0b55dc0 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 -- 172.21.15.63:0/4281601120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 msgr2=0x5575f0b561a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d41c00640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 0x7f7d200a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x5575f0b57360 tx=0x7f7d2c004a90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009ca0 tx=0x7f7d34009a10 comp rx=0 tx=0).stop 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 -- 172.21.15.63:0/4281601120 shutdown_connections 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b63390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b60a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 --2- 172.21.15.63:0/4281601120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b55dc0 0x5575f0b561a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.276+0000 7f7d47586740 1 -- 172.21.15.63:0/4281601120 >> 172.21.15.63:0/4281601120 conn(0x5575f0b522d0 msgr2=0x5575f0b526b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.277+0000 7f7d47586740 1 -- 172.21.15.63:0/4281601120 shutdown_connections 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.277+0000 7f7d47586740 1 -- 172.21.15.63:0/4281601120 wait complete. 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.277+0000 7f7d47586740 1 -- start start 2024-04-25T18:49:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.277+0000 7f7d47586740 1 Processor -- start 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 0x5575f0b5df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b5e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5575f0b639f0 con 0x5575f0b55dc0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5575f0a71310 con 0x5575f0b60fa0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5575f0b63b70 con 0x5575f0b566e0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d47586740 10 client.0 _ll_register_callbacks cb 0x5575f0a3fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d42600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 0x5575f0b5df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d41200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d42600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 0x5575f0b5df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56602/0 (socket says 172.21.15.63:56602) 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d42600640 1 -- 172.21.15.63:0/1548231049 learned_addr learned my addr 172.21.15.63:0/1548231049 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.278+0000 7f7d42600640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b5e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 -- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 msgr2=0x5575f0b5e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b5e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 -- 172.21.15.63:0/1548231049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 msgr2=0x5575f0b5df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 0x5575f0b5df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 -- 172.21.15.63:0/1548231049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f0896620 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d41200640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0fe1a0 tx=0x7f7d2800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e48be43da81db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d27400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28012110 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7d28004050 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d34036030 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7d280122b0 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d47586740 10 client.25657 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.279+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5575f0b6fd10 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d34036030 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f7d28010430 con 0x5575f0b566e0 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 10 client.25657 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 20 client.25657 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0b71240 0x5575f0b73650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 4 client.25657 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575f0acb8b0 con 0x5575f0b71240 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.280+0000 7f7d42600640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0b71240 0x5575f0b73650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.281+0000 7f7d42600640 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0b71240 0x5575f0b73650 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a80260e9c84b81d server_cookie=2927b109b511b6c2 in_seq=0 out_seq=0 2024-04-25T18:49:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.281+0000 7f7d27400640 10 client.25657 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.281+0000 7f7d27400640 1 -- 172.21.15.63:0/1548231049 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5575f0acb8b0 con 0x5575f0b71240 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.281+0000 7f7d27400640 10 client.25657 handle_command_reply: tid=0 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 client.25657 shutdown 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 msgr2=0x5575f0b5e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e480 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c0fe1a0 tx=0x7f7d2800f790 comp rx=0 tx=0).stop 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 shutdown_connections 2024-04-25T18:49:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0b71240 0x5575f0b73650 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b60fa0 0x5575f0b5e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e480 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 --2- 172.21.15.63:0/1548231049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b55dc0 0x5575f0b5df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d27400640 0 client.25657 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 >> 172.21.15.63:0/1548231049 conn(0x5575f0b522d0 msgr2=0x5575f0b58050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 shutdown_connections 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 1 -- 172.21.15.63:0/1548231049 wait complete. 2024-04-25T18:49:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.282+0000 7f7d47586740 20 client.25657 trim_cache size 0 max 16384 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.299+0000 7f7d47586740 1 Processor -- start 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.299+0000 7f7d47586740 1 -- start start 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.299+0000 7f7d47586740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b6c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.299+0000 7f7d47586740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d47586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b60fa0 0x5575f0b78fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d47586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1ae0b0 con 0x5575f0b60fa0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d47586740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1ab950 con 0x5575f0b55dc0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d47586740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1a97a0 con 0x5575f0b566e0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d42600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b6c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d42600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b6c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38996/0 (socket says 172.21.15.63:38996) 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44808/0 (socket says 172.21.15.63:44808) 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d42600640 1 -- 172.21.15.63:0/3842127620 learned_addr learned my addr 172.21.15.63:0/3842127620 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d42600640 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b60fa0 0x5575f0b78fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 -- 172.21.15.63:0/3842127620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 msgr2=0x5575f0b6c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b6c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 -- 172.21.15.63:0/3842127620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b60fa0 msgr2=0x5575f0b78fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b60fa0 0x5575f0b78fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 -- 172.21.15.63:0/3842127620 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575f0b73cd0 con 0x5575f0b566e0 2024-04-25T18:49:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.300+0000 7f7d41200640 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d280108d0 tx=0x7f7d28005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96b24390481cdf64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.301+0000 7f7d32000640 1 -- 172.21.15.63:0/3842127620 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28028030 con 0x5575f0b566e0 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.301+0000 7f7d32000640 1 -- 172.21.15.63:0/3842127620 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d28005560 con 0x5575f0b566e0 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.301+0000 7f7d32000640 1 -- 172.21.15.63:0/3842127620 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d280035f0 con 0x5575f0b566e0 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.301+0000 7f7d47586740 1 -- 172.21.15.63:0/3842127620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 msgr2=0x5575f0b5e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.301+0000 7f7d47586740 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7d280108d0 tx=0x7f7d28005970 comp rx=0 tx=0).stop 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 -- 172.21.15.63:0/3842127620 shutdown_connections 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b60fa0 0x5575f0b78fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b566e0 0x5575f0b5e7e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 --2- 172.21.15.63:0/3842127620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b6c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 -- 172.21.15.63:0/3842127620 >> 172.21.15.63:0/3842127620 conn(0x5575f0b522d0 msgr2=0x5575f0b4d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 -- 172.21.15.63:0/3842127620 shutdown_connections 2024-04-25T18:49:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 -- 172.21.15.63:0/3842127620 wait complete. 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 -- start start 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.302+0000 7f7d47586740 1 Processor -- start 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b69310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b60fa0 0x5575f0b66590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1a9e10 con 0x5575f0b566e0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1ab280 con 0x5575f0b55dc0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c1aa4e0 con 0x5575f0b60fa0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d47586740 10 client.0 _ll_register_callbacks cb 0x7f7d28012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d42600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d41200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b69310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d42600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39008/0 (socket says 172.21.15.63:39008) 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d42600640 1 -- 172.21.15.63:0/4059724122 learned_addr learned my addr 172.21.15.63:0/4059724122 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.303+0000 7f7d42600640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b60fa0 0x5575f0b66590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 -- 172.21.15.63:0/4059724122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b60fa0 msgr2=0x5575f0b66590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b60fa0 0x5575f0b66590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 -- 172.21.15.63:0/4059724122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 msgr2=0x5575f0b69310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b69310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 -- 172.21.15.63:0/4059724122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575f0b66d90 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d42600640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7d34003d10 tx=0x7f7d34009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70615d94cf0a0797 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d27400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031720 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8543+0+0 (secure 0 0 0) 0x7f7d340318c0 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d28005560 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7d34059130 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d47586740 10 client.25643 fetch_fsmap learned FSMap version 26 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.304+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5575f0bd74f0 con 0x5575f0b55dc0 2024-04-25T18:49:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d28005560 con 0x5575f0b55dc0 2024-04-25T18:49:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 26) ==== 8350+0+0 (secure 0 0 0) 0x7f7d34043070 con 0x5575f0b55dc0 2024-04-25T18:49:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 10 client.25643 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T18:49:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 10 client.25643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:49:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 20 client.25643 populate_metadata read hostname 'smithi063' 2024-04-25T18:49:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0bd79f0 0x5575f0bd9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:49:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 4 client.25643 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:49:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575f0ac5320 con 0x5575f0bd79f0 2024-04-25T18:49:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.305+0000 7f7d41200640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0bd79f0 0x5575f0bd9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:49:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.306+0000 7f7d41200640 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0bd79f0 0x5575f0bd9e10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f47e45574a07503 server_cookie=eb3272f9cd6eac02 in_seq=0 out_seq=0 2024-04-25T18:49:54.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.306+0000 7f7d27400640 10 client.25643 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.367+0000 7f7d27400640 1 -- 172.21.15.63:0/4059724122 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575f0ac5320 con 0x5575f0bd79f0 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.367+0000 7f7d27400640 10 client.25643 handle_command_reply: tid=0 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.367+0000 7f7d47586740 1 client.25643 shutdown 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.367+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 msgr2=0x5575f0b68dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.367+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7d34003d10 tx=0x7f7d34009360 comp rx=0 tx=0).stop 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 shutdown_connections 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5575f0bd79f0 0x5575f0bd9e10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5575f0b60fa0 0x5575f0b66590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575f0b566e0 0x5575f0b69310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d27400640 0 client.25643 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 --2- 172.21.15.63:0/4059724122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5575f0b55dc0 0x5575f0b68dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 >> 172.21.15.63:0/4059724122 conn(0x5575f0b522d0 msgr2=0x5575f0b57770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 shutdown_connections 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 1 -- 172.21.15.63:0/4059724122 wait complete. 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.368+0000 7f7d47586740 20 client.25643 trim_cache size 0 max 16384 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:49:54.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 msgr2=0x7f7d200a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 0x7f7d200a61d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x5575f0b57360 tx=0x7f7d2c004a90 comp rx=0 tx=0).stop 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 msgr2=0x7f7d3c1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e9f0 tx=0x7f7d2c00edf0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 shutdown_connections 2024-04-25T18:49:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7d200a3d10 0x7f7d200a61d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d3c19ce20 0x7f7d3c19d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7d3c107090 0x7f7d3c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 --2- 172.21.15.63:0/2085829929 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7d3c101fc0 0x7f7d3c1985b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 >> 172.21.15.63:0/2085829929 conn(0x7f7d3c0fd8a0 msgr2=0x7f7d3c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 shutdown_connections 2024-04-25T18:49:54.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:49:54.369+0000 7f7d43000640 1 -- 172.21.15.63:0/2085829929 wait complete. 2024-04-25T18:49:54.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 57179 2.89 MB/sec warmup 113 sec latency 18.625 ms 2024-04-25T18:49:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:54 smithi063 ceph-mon[28006]: pgmap v103: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:54 smithi070 ceph-mon[32820]: pgmap v103: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:54.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:54 smithi114 ceph-mon[32988]: pgmap v103: 97 pgs: 97 active+clean; 36 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 654 KiB/s wr, 115 op/s 2024-04-25T18:49:55.317 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:49:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setuser ceph since I am not root 2024-04-25T18:49:55.317 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:49:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setgroup ceph since I am not root 2024-04-25T18:49:55.317 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:49:54 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/1933076828 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556a85fcfa40 con 0x556a85fa2800 2024-04-25T18:49:55.317 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:49:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T18:49:55.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 58339 2.91 MB/sec warmup 114 sec latency 17.268 ms 2024-04-25T18:49:55.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:55 smithi063 ceph-mon[28006]: daemon mds.f restarted 2024-04-25T18:49:55.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:55 smithi070 ceph-mon[32820]: daemon mds.f restarted 2024-04-25T18:49:55.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:55 smithi114 ceph-mon[32988]: daemon mds.f restarted 2024-04-25T18:49:56.444 INFO:tasks.workunit.client.0.smithi063.stdout: 1 60034 2.96 MB/sec warmup 115 sec latency 59.170 ms 2024-04-25T18:49:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: pgmap v104: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 181 op/s 2024-04-25T18:49:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T18:49:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/1965078311,v1:172.21.15.114:6835/1965078311] up:boot 2024-04-25T18:49:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:49:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:49:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:56 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: pgmap v104: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 181 op/s 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/1965078311,v1:172.21.15.114:6835/1965078311] up:boot 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:49:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:56 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: pgmap v104: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.0 MiB/s wr, 181 op/s 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/1965078311,v1:172.21.15.114:6835/1965078311] up:boot 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:49:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:56 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:49:57.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 61083 2.99 MB/sec warmup 116 sec latency 16.547 ms 2024-04-25T18:49:58.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 62770 3.02 MB/sec warmup 117 sec latency 17.237 ms 2024-04-25T18:49:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:49:58 smithi063 ceph-mon[28006]: pgmap v106: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 980 KiB/s wr, 159 op/s 2024-04-25T18:49:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:49:58 smithi070 ceph-mon[32820]: pgmap v106: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 980 KiB/s wr, 159 op/s 2024-04-25T18:49:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:49:58 smithi114 ceph-mon[32988]: pgmap v106: 97 pgs: 97 active+clean; 40 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 980 KiB/s wr, 159 op/s 2024-04-25T18:49:59.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 64181 3.09 MB/sec warmup 118 sec latency 14.089 ms 2024-04-25T18:50:00.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 64743 3.07 MB/sec warmup 119 sec latency 325.567 ms 2024-04-25T18:50:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:00 smithi063 ceph-mon[28006]: pgmap v107: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:00.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:00 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T18:50:00.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:00 smithi070 ceph-mon[32820]: pgmap v107: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:00.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:00 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T18:50:00.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:00 smithi114 ceph-mon[32988]: pgmap v107: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:00 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T18:50:02.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 66630 9.20 MB/sec execute 1 sec latency 774.647 ms 2024-04-25T18:50:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:02 smithi070 ceph-mon[32820]: pgmap v108: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:02 smithi114 ceph-mon[32988]: pgmap v108: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:02 smithi063 ceph-mon[28006]: pgmap v108: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:03.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 67788 8.66 MB/sec execute 2 sec latency 15.785 ms 2024-04-25T18:50:04.445 INFO:tasks.workunit.client.0.smithi063.stdout: 1 68832 7.46 MB/sec execute 3 sec latency 21.297 ms 2024-04-25T18:50:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:04 smithi070 ceph-mon[32820]: pgmap v109: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:04.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:04 smithi114 ceph-mon[32988]: pgmap v109: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:04 smithi063 ceph-mon[28006]: pgmap v109: 97 pgs: 97 active+clean; 35 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 991 KiB/s wr, 207 op/s 2024-04-25T18:50:05.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 70401 7.47 MB/sec execute 4 sec latency 15.978 ms 2024-04-25T18:50:06.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 71426 7.29 MB/sec execute 5 sec latency 15.283 ms 2024-04-25T18:50:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:06 smithi070 ceph-mon[32820]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 227 op/s 2024-04-25T18:50:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:06 smithi114 ceph-mon[32988]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 227 op/s 2024-04-25T18:50:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:06 smithi063 ceph-mon[28006]: pgmap v110: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 227 op/s 2024-04-25T18:50:07.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 72456 6.92 MB/sec execute 6 sec latency 49.224 ms 2024-04-25T18:50:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 -- 172.21.15.63:0/1918128512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440102990 msgr2=0x7f44401069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 --2- 172.21.15.63:0/1918128512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440102990 0x7f44401069d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4428009be0 tx=0x7f442802a6c0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 -- 172.21.15.63:0/1918128512 shutdown_connections 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 --2- 172.21.15.63:0/1918128512 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440102990 0x7f44401069d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 --2- 172.21.15.63:0/1918128512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 0x7f444010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 --2- 172.21.15.63:0/1918128512 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f44401023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 -- 172.21.15.63:0/1918128512 >> 172.21.15.63:0/1918128512 conn(0x7f44400fd8a0 msgr2=0x7f44400ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 -- 172.21.15.63:0/1918128512 shutdown_connections 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.523+0000 7f443f200640 1 -- 172.21.15.63:0/1918128512 wait complete. 2024-04-25T18:50:07.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.524+0000 7f443f200640 1 Processor -- start 2024-04-25T18:50:07.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 -- start start 2024-04-25T18:50:07.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 0x7f4440074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440074cd0 0x7f44401a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4440111fd0 con 0x7f44401070b0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4440111e50 con 0x7f4440074cd0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443f200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4440112150 con 0x7f4440101fe0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40146/0 (socket says 172.21.15.63:40146) 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443de00640 1 -- 172.21.15.63:0/3036071137 learned_addr learned my addr 172.21.15.63:0/3036071137 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443d400640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 0x7f4440074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.525+0000 7f443e800640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440074cd0 0x7f44401a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 -- 172.21.15.63:0/3036071137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440074cd0 msgr2=0x7f44401a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440074cd0 0x7f44401a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 -- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 msgr2=0x7f4440074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 0x7f4440074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 -- 172.21.15.63:0/3036071137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44401a6ea0 con 0x7f4440101fe0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443de00640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f443800e9c0 tx=0x7f443800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f4436a00640 1 -- 172.21.15.63:0/3036071137 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443800cde0 con 0x7f4440101fe0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f4436a00640 1 -- 172.21.15.63:0/3036071137 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44380035f0 con 0x7f4440101fe0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.526+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44401a7190 con 0x7f4440101fe0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.527+0000 7f4436a00640 1 -- 172.21.15.63:0/3036071137 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4438010690 con 0x7f4440101fe0 2024-04-25T18:50:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.527+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44401a76b0 con 0x7f4440101fe0 2024-04-25T18:50:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.530+0000 7f4447b9d740 1 Processor -- start 2024-04-25T18:50:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 -- start start 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 0x561bc1324d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc132f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4436a00640 1 -- 172.21.15.63:0/3036071137 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4438004000 con 0x7f4440101fe0 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bc128a900 con 0x561bc13249b0 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bc1230ef0 con 0x561bc132fb90 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f4447b9d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bc12868c0 con 0x561bc13252d0 2024-04-25T18:50:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.531+0000 7f443e800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 0x561bc1324d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42502/0 (socket says 172.21.15.63:42502) 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 -- 172.21.15.63:0/2527631036 learned_addr learned my addr 172.21.15.63:0/2527631036 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443de00640 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc132f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 -- 172.21.15.63:0/2527631036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 msgr2=0x561bc132f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc132f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 -- 172.21.15.63:0/2527631036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 msgr2=0x561bc1324d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 0x561bc1324d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 -- 172.21.15.63:0/2527631036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc0f9d210 con 0x561bc132fb90 2024-04-25T18:50:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443e800640 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 0x561bc1324d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.532+0000 7f443d400640 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f444006b3b0 tx=0x7f4428000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=999da69013d28da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f442f000640 1 -- 172.21.15.63:0/2527631036 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428041030 con 0x561bc132fb90 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f442f000640 1 -- 172.21.15.63:0/2527631036 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4428003e20 con 0x561bc132fb90 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f442f000640 1 -- 172.21.15.63:0/2527631036 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428036040 con 0x561bc132fb90 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f4436a00640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 0x7f44240a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f4436a00640 1 -- 172.21.15.63:0/3036071137 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f44380d0b10 con 0x7f4440101fe0 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.533+0000 7f443e800640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 0x7f44240a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 -- 172.21.15.63:0/2527631036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 msgr2=0x561bc1331f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f444006b3b0 tx=0x7f4428000f40 comp rx=0 tx=0).stop 2024-04-25T18:50:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 -- 172.21.15.63:0/2527631036 shutdown_connections 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc132fb90 0x561bc1331f80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc132f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2527631036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc13249b0 0x561bc1324d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 -- 172.21.15.63:0/2527631036 >> 172.21.15.63:0/2527631036 conn(0x561bc1320ec0 msgr2=0x561bc13212a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 -- 172.21.15.63:0/2527631036 shutdown_connections 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f4447b9d740 1 -- 172.21.15.63:0/2527631036 wait complete. 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.534+0000 7f443e800640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 0x7f44240a61d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4430009e10 tx=0x7f4430005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.535+0000 7f4447b9d740 1 -- start start 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.535+0000 7f4447b9d740 1 Processor -- start 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.535+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bc1332760 con 0x561bc132fb90 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bc1244310 con 0x561bc13252d0 2024-04-25T18:50:07.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bc13328e0 con 0x561bc13249b0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f4447b9d740 10 client.0 _ll_register_callbacks cb 0x561bc1212cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42510/0 (socket says 172.21.15.63:42510) 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40152/0 (socket says 172.21.15.63:40152) 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 -- 172.21.15.63:0/2537133336 learned_addr learned my addr 172.21.15.63:0/2537133336 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443e800640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 -- 172.21.15.63:0/2537133336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 msgr2=0x561bc132ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 -- 172.21.15.63:0/2537133336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 msgr2=0x561bc132d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443d400640 1 -- 172.21.15.63:0/2537133336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc1069620 con 0x561bc13252d0 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443de00640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.536+0000 7f443e800640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:07.540 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.537+0000 7f443d400640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4428009be0 tx=0x7f4428009940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c73876e5fb2d6d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.537+0000 7f4423400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.537+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4428036050 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.537+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f4428030970 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.537+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4428003e20 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4428041040 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 10 client.25661 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bc133e920 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4428003e20 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f44280045d0 con 0x561bc13252d0 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 10 client.25661 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 20 client.25661 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:07.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc133fc70 0x561bc1342060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 4 client.25661 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.538+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc129e8b0 con 0x561bc133fc70 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.539+0000 7f443de00640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc133fc70 0x561bc1342060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.539+0000 7f443de00640 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc133fc70 0x561bc1342060 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea6efb728ee4dc4 server_cookie=9072422438e22bd in_seq=0 out_seq=0 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.539+0000 7f4423400640 10 client.25661 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4423400640 1 -- 172.21.15.63:0/2537133336 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bc129e8b0 con 0x561bc133fc70 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4423400640 10 client.25661 handle_command_reply: tid=0 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 client.25661 shutdown 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 msgr2=0x561bc132cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4428009be0 tx=0x7f4428009940 comp rx=0 tx=0).stop 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 shutdown_connections 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc133fc70 0x561bc1342060 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc132cfd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2537133336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc132ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 >> 172.21.15.63:0/2537133336 conn(0x561bc1320ec0 msgr2=0x561bc1326c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.540+0000 7f4423400640 0 client.25661 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.541+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 shutdown_connections 2024-04-25T18:50:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.541+0000 7f4447b9d740 1 -- 172.21.15.63:0/2537133336 wait complete. 2024-04-25T18:50:07.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.541+0000 7f4447b9d740 20 client.25661 trim_cache size 0 max 16384 2024-04-25T18:50:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.560+0000 7f4447b9d740 1 Processor -- start 2024-04-25T18:50:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.560+0000 7f4447b9d740 1 -- start start 2024-04-25T18:50:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.560+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.560+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc131fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f4447b9d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f44401b2600 con 0x561bc132fb90 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f4447b9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44401afe40 con 0x561bc13249b0 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f4447b9d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44401adcb0 con 0x561bc13252d0 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443e800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42514/0 (socket says 172.21.15.63:42514) 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 -- 172.21.15.63:0/2671440179 learned_addr learned my addr 172.21.15.63:0/2671440179 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443d400640 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc131fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 -- 172.21.15.63:0/2671440179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 msgr2=0x561bc131fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc131fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 -- 172.21.15.63:0/2671440179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 msgr2=0x561bc132ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.561+0000 7f443de00640 1 -- 172.21.15.63:0/2671440179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc1320cf0 con 0x561bc13249b0 2024-04-25T18:50:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.562+0000 7f443de00640 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f443809ca30 tx=0x7f4438087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0bd77aa0b20104 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.562+0000 7f442e600640 1 -- 172.21.15.63:0/2671440179 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44380d5040 con 0x561bc13249b0 2024-04-25T18:50:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.562+0000 7f442e600640 1 -- 172.21.15.63:0/2671440179 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f443809d1a0 con 0x561bc13249b0 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.562+0000 7f442e600640 1 -- 172.21.15.63:0/2671440179 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44380bd0d0 con 0x561bc13249b0 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- 172.21.15.63:0/2671440179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 msgr2=0x561bc1327370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f443809ca30 tx=0x7f4438087040 comp rx=0 tx=0).stop 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- 172.21.15.63:0/2671440179 shutdown_connections 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc132ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13252d0 0x561bc131fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 --2- 172.21.15.63:0/2671440179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13249b0 0x561bc1327370 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- 172.21.15.63:0/2671440179 >> 172.21.15.63:0/2671440179 conn(0x561bc1320ec0 msgr2=0x561bc131f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- 172.21.15.63:0/2671440179 shutdown_connections 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- 172.21.15.63:0/2671440179 wait complete. 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.563+0000 7f4447b9d740 1 -- start start 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 Processor -- start 2024-04-25T18:50:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc13a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f44401ae300 con 0x561bc132fb90 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44401af770 con 0x561bc13252d0 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44401ae9d0 con 0x561bc13249b0 2024-04-25T18:50:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f4447b9d740 10 client.0 _ll_register_callbacks cb 0x7f4438005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443e800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc13a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40172/0 (socket says 172.21.15.63:40172) 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443de00640 1 -- 172.21.15.63:0/3579515070 learned_addr learned my addr 172.21.15.63:0/3579515070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.564+0000 7f443d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42520/0 (socket says 172.21.15.63:42520) 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 -- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 msgr2=0x561bc13a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 -- 172.21.15.63:0/3579515070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 msgr2=0x561bc13a3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc13a3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 -- 172.21.15.63:0/3579515070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc13ad6c0 con 0x561bc13249b0 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443d400640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:50:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f443de00640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4438098f00 tx=0x7f44380d4660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18a3670d3cf80d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f4423400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.565+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443809c400 con 0x561bc13249b0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f443809ada0 con 0x561bc13249b0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f443809d1a0 con 0x561bc13249b0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44380042f0 con 0x561bc13249b0 2024-04-25T18:50:07.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 10 client.25687 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561bc13ae9d0 con 0x561bc13249b0 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f443809d1a0 con 0x561bc13249b0 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f4438099110 con 0x561bc13249b0 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 10 client.25687 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 10 client.25687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 20 client.25687 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc13af090 0x561bc13b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 4 client.25687 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.566+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561bc1298320 con 0x561bc13af090 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.567+0000 7f443d400640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc13af090 0x561bc13b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.567+0000 7f443d400640 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc13af090 0x561bc13b1480 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3af953c621f18aac server_cookie=c9decfdddeca0fbb in_seq=0 out_seq=0 2024-04-25T18:50:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.567+0000 7f4423400640 10 client.25687 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4423400640 1 -- 172.21.15.63:0/3579515070 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561bc1298320 con 0x561bc13af090 2024-04-25T18:50:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4423400640 10 client.25687 handle_command_reply: tid=0 2024-04-25T18:50:07.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 client.25687 shutdown 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 msgr2=0x561bc13a25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4438098f00 tx=0x7f44380d4660 comp rx=0 tx=0).stop 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 shutdown_connections 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bc13af090 0x561bc13b1480 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bc13252d0 0x561bc13a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bc132fb90 0x561bc13a3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 --2- 172.21.15.63:0/3579515070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bc13249b0 0x561bc13a25f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4423400640 0 client.25687 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 >> 172.21.15.63:0/3579515070 conn(0x561bc1320ec0 msgr2=0x561bc133ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 shutdown_connections 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.612+0000 7f4447b9d740 1 -- 172.21.15.63:0/3579515070 wait complete. 2024-04-25T18:50:07.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.613+0000 7f4447b9d740 20 client.25687 trim_cache size 0 max 16384 2024-04-25T18:50:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:50:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:50:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:50:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:50:07.615 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.613+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 msgr2=0x7f44240a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.613+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 0x7f44240a61d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4430009e10 tx=0x7f4430005f90 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.613+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 msgr2=0x7f4440074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.613+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f443800e9c0 tx=0x7f443800ee80 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 shutdown_connections 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44240a3d10 0x7f44240a61d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4440074cd0 0x7f44401a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44401070b0 0x7f4440074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 --2- 172.21.15.63:0/3036071137 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4440101fe0 0x7f4440074250 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 >> 172.21.15.63:0/3036071137 conn(0x7f44400fd8a0 msgr2=0x7f44400fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 shutdown_connections 2024-04-25T18:50:07.616 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:07.614+0000 7f443f200640 1 -- 172.21.15.63:0/3036071137 wait complete. 2024-04-25T18:50:08.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 73924 7.01 MB/sec execute 7 sec latency 16.119 ms 2024-04-25T18:50:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:08 smithi070 ceph-mon[32820]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 861 KiB/s wr, 195 op/s 2024-04-25T18:50:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:08 smithi114 ceph-mon[32988]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 861 KiB/s wr, 195 op/s 2024-04-25T18:50:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:08 smithi063 ceph-mon[28006]: pgmap v111: 97 pgs: 97 active+clean; 38 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 861 KiB/s wr, 195 op/s 2024-04-25T18:50:09.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 74932 6.95 MB/sec execute 8 sec latency 18.348 ms 2024-04-25T18:50:10.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 75945 6.74 MB/sec execute 9 sec latency 21.912 ms 2024-04-25T18:50:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:10 smithi070 ceph-mon[32820]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 273 op/s 2024-04-25T18:50:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:10 smithi114 ceph-mon[32988]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 273 op/s 2024-04-25T18:50:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:10 smithi063 ceph-mon[28006]: pgmap v112: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 273 op/s 2024-04-25T18:50:11.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 77473 6.82 MB/sec execute 10 sec latency 17.041 ms 2024-04-25T18:50:12.446 INFO:tasks.workunit.client.0.smithi063.stdout: 1 78469 6.79 MB/sec execute 11 sec latency 22.028 ms 2024-04-25T18:50:12.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:12 smithi114 ceph-mon[32988]: pgmap v113: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:12 smithi070 ceph-mon[32820]: pgmap v113: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:12 smithi063 ceph-mon[28006]: pgmap v113: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:13.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 79521 6.65 MB/sec execute 12 sec latency 32.563 ms 2024-04-25T18:50:14.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 81058 6.72 MB/sec execute 13 sec latency 17.111 ms 2024-04-25T18:50:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:14 smithi070 ceph-mon[32820]: pgmap v114: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:14.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:14 smithi114 ceph-mon[32988]: pgmap v114: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:14 smithi063 ceph-mon[28006]: pgmap v114: 97 pgs: 97 active+clean; 38 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 806 KiB/s wr, 165 op/s 2024-04-25T18:50:15.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 82057 6.70 MB/sec execute 14 sec latency 17.539 ms 2024-04-25T18:50:16.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 83060 6.59 MB/sec execute 15 sec latency 23.502 ms 2024-04-25T18:50:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:16 smithi070 ceph-mon[32820]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2024-04-25T18:50:16.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:16 smithi114 ceph-mon[32988]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2024-04-25T18:50:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:16 smithi063 ceph-mon[28006]: pgmap v115: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.4 MiB/s wr, 226 op/s 2024-04-25T18:50:17.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 84655 6.65 MB/sec execute 16 sec latency 46.701 ms 2024-04-25T18:50:18.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 85887 6.66 MB/sec execute 17 sec latency 23.731 ms 2024-04-25T18:50:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:18 smithi070 ceph-mon[32820]: pgmap v116: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1020 KiB/s wr, 144 op/s 2024-04-25T18:50:18.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:18 smithi114 ceph-mon[32988]: pgmap v116: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1020 KiB/s wr, 144 op/s 2024-04-25T18:50:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:18 smithi063 ceph-mon[28006]: pgmap v116: 97 pgs: 97 active+clean; 43 MiB data, 661 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1020 KiB/s wr, 144 op/s 2024-04-25T18:50:19.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 87554 6.67 MB/sec execute 18 sec latency 14.006 ms 2024-04-25T18:50:20.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 89009 6.94 MB/sec execute 19 sec latency 14.422 ms 2024-04-25T18:50:20.680 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:20 smithi063 ceph-mon[28006]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-04-25T18:50:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:20 smithi070 ceph-mon[32820]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 -- 172.21.15.63:0/2115497462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff274100af0 msgr2=0x7ff274100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 --2- 172.21.15.63:0/2115497462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff274100af0 0x7ff274100ed0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff264009900 tx=0x7ff26402a410 comp rx=0 tx=0).stop 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 -- 172.21.15.63:0/2115497462 shutdown_connections 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 --2- 172.21.15.63:0/2115497462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff27410e6a0 0x7ff274110a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 --2- 172.21.15.63:0/2115497462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff27410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 --2- 172.21.15.63:0/2115497462 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff274100af0 0x7ff274100ed0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 -- 172.21.15.63:0/2115497462 >> 172.21.15.63:0/2115497462 conn(0x7ff2740fc770 msgr2=0x7ff2740feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 -- 172.21.15.63:0/2115497462 shutdown_connections 2024-04-25T18:50:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.766+0000 7ff273e00640 1 -- 172.21.15.63:0/2115497462 wait complete. 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.767+0000 7ff273e00640 1 Processor -- start 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 -- start start 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff2741a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27410e6a0 0x7ff27419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff274114240 con 0x7ff2741014a0 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741140c0 con 0x7ff274100af0 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741143c0 con 0x7ff27410e6a0 2024-04-25T18:50:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff272a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff272a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44314/0 (socket says 172.21.15.63:44314) 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff272a00640 1 -- 172.21.15.63:0/1875176575 learned_addr learned my addr 172.21.15.63:0/1875176575 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff273400640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27410e6a0 0x7ff27419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.768+0000 7ff272000640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff2741a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 -- 172.21.15.63:0/1875176575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27410e6a0 msgr2=0x7ff27419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27410e6a0 0x7ff27419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 -- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 msgr2=0x7ff2741a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff2741a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 -- 172.21.15.63:0/1875176575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27419af40 con 0x7ff274100af0 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272000640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff2741a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff272a00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff264005ab0 tx=0x7ff264003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff26b400640 1 -- 172.21.15.63:0/1875176575 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26403f070 con 0x7ff274100af0 2024-04-25T18:50:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27419b1d0 con 0x7ff274100af0 2024-04-25T18:50:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff26b400640 1 -- 172.21.15.63:0/1875176575 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff264031d60 con 0x7ff274100af0 2024-04-25T18:50:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.769+0000 7ff26b400640 1 -- 172.21.15.63:0/1875176575 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264035030 con 0x7ff274100af0 2024-04-25T18:50:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.770+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2741a7520 con 0x7ff274100af0 2024-04-25T18:50:20.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 Processor -- start 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 -- start start 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff26b400640 1 -- 172.21.15.63:0/1875176575 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff264042070 con 0x7ff274100af0 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 0x5562dba06d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba072d0 0x5562dba11650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562db96c900 con 0x5562dba072d0 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562db912ef0 con 0x5562dba069b0 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.773+0000 7ff27c9de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562db9688c0 con 0x5562dba11b90 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff272000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff272a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba072d0 0x5562dba11650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 0x5562dba06d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff272000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33842/0 (socket says 172.21.15.63:33842) 2024-04-25T18:50:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff272000640 1 -- 172.21.15.63:0/311887494 learned_addr learned my addr 172.21.15.63:0/311887494 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 -- 172.21.15.63:0/311887494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 msgr2=0x5562dba13f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 -- 172.21.15.63:0/311887494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba072d0 msgr2=0x5562dba11650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba072d0 0x5562dba11650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff273400640 1 -- 172.21.15.63:0/311887494 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562db67f210 con 0x5562dba069b0 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.774+0000 7ff272000640 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff273400640 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 0x5562dba06d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff26c005fd0 tx=0x7ff26c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9ddeb3949883d8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff25fe00640 1 -- 172.21.15.63:0/311887494 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c004d20 con 0x5562dba069b0 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff25fe00640 1 -- 172.21.15.63:0/311887494 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff26c004050 con 0x5562dba069b0 2024-04-25T18:50:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff25fe00640 1 -- 172.21.15.63:0/311887494 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c005280 con 0x5562dba069b0 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff26b400640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 0x7ff2580a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff26b400640 1 -- 172.21.15.63:0/1875176575 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7ff2640f0d40 con 0x7ff274100af0 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff272a00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 0x7ff2580a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff27c9de740 1 -- 172.21.15.63:0/311887494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 msgr2=0x5562dba06d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.775+0000 7ff27c9de740 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 0x5562dba06d90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff26c005fd0 tx=0x7ff26c011810 comp rx=0 tx=0).stop 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 -- 172.21.15.63:0/311887494 shutdown_connections 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba13f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba072d0 0x5562dba11650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 --2- 172.21.15.63:0/311887494 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba069b0 0x5562dba06d90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 -- 172.21.15.63:0/311887494 >> 172.21.15.63:0/311887494 conn(0x5562dba02ec0 msgr2=0x5562dba032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 -- 172.21.15.63:0/311887494 shutdown_connections 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 -- 172.21.15.63:0/311887494 wait complete. 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff272a00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 0x7ff2580a61d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5562dba07f50 tx=0x7ff2640f47f0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.776+0000 7ff27c9de740 1 -- start start 2024-04-25T18:50:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 Processor -- start 2024-04-25T18:50:20.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 0x5562dba0eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba11b90 0x5562dba0f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5562dba147c0 con 0x5562dba11b90 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5562db926310 con 0x5562dba072d0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5562dba14940 con 0x5562dba069b0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff27c9de740 10 client.0 _ll_register_callbacks cb 0x5562db8f4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff273400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 0x5562dba0eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.777+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44332/0 (socket says 172.21.15.63:44332) 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff272000640 1 -- 172.21.15.63:0/1367659692 learned_addr learned my addr 172.21.15.63:0/1367659692 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba11b90 0x5562dba0f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 -- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 msgr2=0x5562dba0f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 -- 172.21.15.63:0/1367659692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba11b90 msgr2=0x5562dba0f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba11b90 0x5562dba0f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 -- 172.21.15.63:0/1367659692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562db74b620 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff272000640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff273400640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 0x5562dba0eb00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff26c01b040 tx=0x7ff26c009a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae257b41850e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.778+0000 7ff25cc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26c003d30 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7ff26c028480 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff26c004050 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff26c036050 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff27c9de740 10 client.25699 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562dba20a70 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff26c004050 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff26c025040 con 0x5562dba069b0 2024-04-25T18:50:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff27c9de740 10 client.25699 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.779+0000 7ff27c9de740 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff27c9de740 20 client.25699 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba24260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff27c9de740 4 client.25699 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5562db9808b0 con 0x5562dba21e70 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff272000640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba24260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff272000640 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba24260 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed9aa51ed15f23f server_cookie=6558847a17626977 in_seq=0 out_seq=0 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.780+0000 7ff25cc00640 10 client.25699 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.781+0000 7ff25cc00640 1 -- 172.21.15.63:0/1367659692 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5562db9808b0 con 0x5562dba21e70 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.781+0000 7ff25cc00640 10 client.25699 handle_command_reply: tid=0 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.781+0000 7ff27c9de740 1 client.25699 shutdown 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.781+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 msgr2=0x5562dba0eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.781+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 0x5562dba0eb00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff26c01b040 tx=0x7ff26c009a50 comp rx=0 tx=0).stop 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 shutdown_connections 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba24260 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba0f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba11b90 0x5562dba0f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff25cc00640 0 client.25699 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1367659692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba069b0 0x5562dba0eb00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 >> 172.21.15.63:0/1367659692 conn(0x5562dba02ec0 msgr2=0x5562dba08c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 shutdown_connections 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 1 -- 172.21.15.63:0/1367659692 wait complete. 2024-04-25T18:50:20.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.782+0000 7ff27c9de740 20 client.25699 trim_cache size 0 max 16384 2024-04-25T18:50:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.799+0000 7ff27c9de740 1 Processor -- start 2024-04-25T18:50:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 -- start start 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba1d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba29f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741b2590 con 0x5562dba069b0 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741afdd0 con 0x5562dba072d0 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff27c9de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741adc20 con 0x5562dba11b90 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44348/0 (socket says 172.21.15.63:44348) 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 -- 172.21.15.63:0/1353376006 learned_addr learned my addr 172.21.15.63:0/1353376006 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff273400640 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba1d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 -- 172.21.15.63:0/1353376006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 msgr2=0x5562dba29f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba29f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 -- 172.21.15.63:0/1353376006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 msgr2=0x5562dba1d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.800+0000 7ff272000640 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba1d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff272000640 1 -- 172.21.15.63:0/1353376006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5562dba1eb80 con 0x5562dba072d0 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff273400640 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba1d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff272000640 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff260005e00 tx=0x7ff2600044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c167497cc2794a5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff25f400640 1 -- 172.21.15.63:0/1353376006 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26001d430 con 0x5562dba072d0 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff25f400640 1 -- 172.21.15.63:0/1353376006 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff260011440 con 0x5562dba072d0 2024-04-25T18:50:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.801+0000 7ff25f400640 1 -- 172.21.15.63:0/1353376006 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260010630 con 0x5562dba072d0 2024-04-25T18:50:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- 172.21.15.63:0/1353376006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 msgr2=0x5562dba29a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff260005e00 tx=0x7ff2600044d0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- 172.21.15.63:0/1353376006 shutdown_connections 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba29a20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba29f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1353376006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba1d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- 172.21.15.63:0/1353376006 >> 172.21.15.63:0/1353376006 conn(0x5562dba02ec0 msgr2=0x5562dba08360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- 172.21.15.63:0/1353376006 shutdown_connections 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- 172.21.15.63:0/1353376006 wait complete. 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.802+0000 7ff27c9de740 1 -- start start 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 Processor -- start 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba19c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741ae290 con 0x5562dba069b0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741af700 con 0x5562dba072d0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff2741ae960 con 0x5562dba11b90 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff27c9de740 10 client.0 _ll_register_callbacks cb 0x7ff26c004de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.803+0000 7ff272000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44354/0 (socket says 172.21.15.63:44354) 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 -- 172.21.15.63:0/1092218787 learned_addr learned my addr 172.21.15.63:0/1092218787 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff273400640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff273400640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba19c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 -- 172.21.15.63:0/1092218787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 msgr2=0x5562dba171a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba171a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 -- 172.21.15.63:0/1092218787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 msgr2=0x5562dba19c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba19c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 -- 172.21.15.63:0/1092218787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562dba179a0 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff273400640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba19c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff272000640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff260006010 tx=0x7ff260003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daaba7a38c6a908d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff25cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.804+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff26001d940 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7ff2600115f0 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff260011440 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff2600117f0 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 10 client.25685 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5562dba87eb0 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff260011440 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7ff260027410 con 0x5562dba072d0 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 10 client.25685 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 10 client.25685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 20 client.25685 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba8a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 4 client.25685 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.805+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5562db97a320 con 0x5562dba21e70 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.806+0000 7ff273400640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba8a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.806+0000 7ff273400640 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba8a5a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a01bd9890860ae06 server_cookie=129f444de1fdfd4d in_seq=0 out_seq=0 2024-04-25T18:50:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.806+0000 7ff25cc00640 10 client.25685 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:20.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:20 smithi114 ceph-mon[32988]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 221 op/s 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff25cc00640 1 -- 172.21.15.63:0/1092218787 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5562db97a320 con 0x5562dba21e70 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff25cc00640 10 client.25685 handle_command_reply: tid=0 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 client.25685 shutdown 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 msgr2=0x5562dba1a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff260006010 tx=0x7ff260003f20 comp rx=0 tx=0).stop 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 shutdown_connections 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5562dba21e70 0x5562dba8a5a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5562dba072d0 0x5562dba1a150 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5562dba11b90 0x5562dba171a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 --2- 172.21.15.63:0/1092218787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5562dba069b0 0x5562dba19c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 >> 172.21.15.63:0/1092218787 conn(0x5562dba02ec0 msgr2=0x5562dba08bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff25cc00640 0 client.25685 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.857+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 shutdown_connections 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.858+0000 7ff27c9de740 1 -- 172.21.15.63:0/1092218787 wait complete. 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.858+0000 7ff27c9de740 20 client.25685 trim_cache size 0 max 16384 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:50:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.858+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 msgr2=0x7ff2580a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.858+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 0x7ff2580a61d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x5562dba07f50 tx=0x7ff2640f47f0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 msgr2=0x7ff2741a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff264005ab0 tx=0x7ff264003c80 comp rx=0 tx=0).stop 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 shutdown_connections 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff2580a3d10 0x7ff2580a61d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff27410e6a0 0x7ff27419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2741014a0 0x7ff2741a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 --2- 172.21.15.63:0/1875176575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff274100af0 0x7ff2741a0530 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 >> 172.21.15.63:0/1875176575 conn(0x7ff2740fc770 msgr2=0x7ff2740feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 shutdown_connections 2024-04-25T18:50:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:20.859+0000 7ff273e00640 1 -- 172.21.15.63:0/1875176575 wait complete. 2024-04-25T18:50:21.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 90181 6.86 MB/sec execute 20 sec latency 31.812 ms 2024-04-25T18:50:22.447 INFO:tasks.workunit.client.0.smithi063.stdout: 1 92073 6.98 MB/sec execute 21 sec latency 35.633 ms 2024-04-25T18:50:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:22 smithi070 ceph-mon[32820]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:22 smithi114 ceph-mon[32988]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:22.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:22 smithi063 ceph-mon[28006]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:23.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 93172 6.91 MB/sec execute 22 sec latency 22.339 ms 2024-04-25T18:50:24.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 95148 7.06 MB/sec execute 23 sec latency 15.041 ms 2024-04-25T18:50:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:24 smithi070 ceph-mon[32820]: pgmap v119: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:24.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:24 smithi114 ceph-mon[32988]: pgmap v119: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:24 smithi063 ceph-mon[28006]: pgmap v119: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 137 op/s 2024-04-25T18:50:25.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 96317 7.10 MB/sec execute 24 sec latency 15.707 ms 2024-04-25T18:50:26.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 97592 7.03 MB/sec execute 25 sec latency 18.994 ms 2024-04-25T18:50:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:26 smithi070 ceph-mon[32820]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-04-25T18:50:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:26 smithi114 ceph-mon[32988]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-04-25T18:50:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:26 smithi063 ceph-mon[28006]: pgmap v120: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2024-04-25T18:50:27.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 99368 7.22 MB/sec execute 26 sec latency 52.434 ms 2024-04-25T18:50:28.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 100318 7.05 MB/sec execute 27 sec latency 26.278 ms 2024-04-25T18:50:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:28 smithi070 ceph-mon[32820]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-04-25T18:50:28.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:28 smithi114 ceph-mon[32988]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-04-25T18:50:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:28 smithi063 ceph-mon[28006]: pgmap v121: 97 pgs: 97 active+clean; 45 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 211 op/s 2024-04-25T18:50:29.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 102134 7.16 MB/sec execute 28 sec latency 17.610 ms 2024-04-25T18:50:30.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 103361 7.19 MB/sec execute 29 sec latency 15.208 ms 2024-04-25T18:50:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:30 smithi070 ceph-mon[32820]: pgmap v122: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-04-25T18:50:30.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:30 smithi114 ceph-mon[32988]: pgmap v122: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-04-25T18:50:30.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:30 smithi063 ceph-mon[28006]: pgmap v122: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-04-25T18:50:31.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 104602 7.13 MB/sec execute 30 sec latency 20.491 ms 2024-04-25T18:50:32.448 INFO:tasks.workunit.client.0.smithi063.stdout: 1 106271 7.20 MB/sec execute 31 sec latency 36.539 ms 2024-04-25T18:50:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:32 smithi070 ceph-mon[32820]: pgmap v123: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:32.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:32 smithi114 ceph-mon[32988]: pgmap v123: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:32.854 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:32 smithi063 ceph-mon[28006]: pgmap v123: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:33.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 107251 7.13 MB/sec execute 32 sec latency 18.803 ms 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 -- 172.21.15.63:0/4046127915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4101880 msgr2=0x7fe3c41058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 --2- 172.21.15.63:0/4046127915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4101880 0x7fe3c41058c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009900 tx=0x7fe3b402a410 comp rx=0 tx=0).stop 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 -- 172.21.15.63:0/4046127915 shutdown_connections 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 --2- 172.21.15.63:0/4046127915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c4101880 0x7fe3c41058c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 --2- 172.21.15.63:0/4046127915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4105fa0 0x7fe3c410db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 --2- 172.21.15.63:0/4046127915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4100ed0 0x7fe3c41012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 -- 172.21.15.63:0/4046127915 >> 172.21.15.63:0/4046127915 conn(0x7fe3c40fc770 msgr2=0x7fe3c40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 -- 172.21.15.63:0/4046127915 shutdown_connections 2024-04-25T18:50:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.020+0000 7fe3cb200640 1 -- 172.21.15.63:0/4046127915 wait complete. 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.021+0000 7fe3cb200640 1 Processor -- start 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 -- start start 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4105fa0 0x7fe3c4102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c41033e0 0x7fe3c41a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4110f50 con 0x7fe3c4105fa0 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c4110dd0 con 0x7fe3c41033e0 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3cb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41110d0 con 0x7fe3c4100ed0 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37172/0 (socket says 172.21.15.63:37172) 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3c9e00640 1 -- 172.21.15.63:0/1265033330 learned_addr learned my addr 172.21.15.63:0/1265033330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3c9400640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4105fa0 0x7fe3c4102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.022+0000 7fe3ca800640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c41033e0 0x7fe3c41a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 -- 172.21.15.63:0/1265033330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c41033e0 msgr2=0x7fe3c41a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c41033e0 0x7fe3c41a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 -- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4105fa0 msgr2=0x7fe3c4102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4105fa0 0x7fe3c4102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 -- 172.21.15.63:0/1265033330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c41a6e90 con 0x7fe3c4100ed0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3c9e00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00ebc0 tx=0x7fe3bc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3baa00640 1 -- 172.21.15.63:0/1265033330 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc010630 con 0x7fe3c4100ed0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3baa00640 1 -- 172.21.15.63:0/1265033330 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3bc010c80 con 0x7fe3c4100ed0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c41a7180 con 0x7fe3c4100ed0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.023+0000 7fe3baa00640 1 -- 172.21.15.63:0/1265033330 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc018b60 con 0x7fe3c4100ed0 2024-04-25T18:50:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.024+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c41a76a0 con 0x7fe3c4100ed0 2024-04-25T18:50:34.028 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.027+0000 7fe3baa00640 1 -- 172.21.15.63:0/1265033330 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe3bc018d00 con 0x7fe3c4100ed0 2024-04-25T18:50:34.028 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.027+0000 7fe3cf5f1740 1 Processor -- start 2024-04-25T18:50:34.028 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.027+0000 7fe3cf5f1740 1 -- start start 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5effd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f0a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f0cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5e65900 con 0x55b8e5f0ab90 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5e0bef0 con 0x55b8e5eff9b0 2024-04-25T18:50:34.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5e618c0 con 0x55b8e5f002d0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f0a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3ca800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5effd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3c9e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f0a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37186/0 (socket says 172.21.15.63:37186) 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3c9e00640 1 -- 172.21.15.63:0/2603463692 learned_addr learned my addr 172.21.15.63:0/2603463692 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.028+0000 7fe3c9400640 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f0cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 -- 172.21.15.63:0/2603463692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 msgr2=0x55b8e5f0a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f0a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 -- 172.21.15.63:0/2603463692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 msgr2=0x55b8e5f0cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f0cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 -- 172.21.15.63:0/2603463692 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e5b78210 con 0x55b8e5eff9b0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3c9400640 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f0cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3ca800640 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5effd90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009b80 tx=0x7fe3c0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11f2ea18fe02ca4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3baa00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 0x7fe3b00a6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3af000640 1 -- 172.21.15.63:0/2603463692 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0004050 con 0x55b8e5eff9b0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3baa00640 1 -- 172.21.15.63:0/1265033330 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe3bc0d0e00 con 0x7fe3c4100ed0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.029+0000 7fe3c9e00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 0x7fe3b00a6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.030+0000 7fe3af000640 1 -- 172.21.15.63:0/2603463692 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3c00054d0 con 0x55b8e5eff9b0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.030+0000 7fe3af000640 1 -- 172.21.15.63:0/2603463692 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0004be0 con 0x55b8e5eff9b0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.030+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2603463692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 msgr2=0x55b8e5effd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5effd90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009b80 tx=0x7fe3c0011810 comp rx=0 tx=0).stop 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3c9e00640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 0x7fe3b00a6200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc099240 tx=0x7fe3bc0d48b0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2603463692 shutdown_connections 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f0cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f0a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2603463692 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5effd90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2603463692 >> 172.21.15.63:0/2603463692 conn(0x55b8e5efbec0 msgr2=0x55b8e5efc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2603463692 shutdown_connections 2024-04-25T18:50:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2603463692 wait complete. 2024-04-25T18:50:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.031+0000 7fe3cf5f1740 1 -- start start 2024-04-25T18:50:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 Processor -- start 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f07b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5f0d7c0 con 0x55b8e5f0ab90 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5e1f310 con 0x55b8e5f002d0 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b8e5f0d940 con 0x55b8e5eff9b0 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.032+0000 7fe3cf5f1740 10 client.0 _ll_register_callbacks cb 0x55b8e5dedcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42034/0 (socket says 172.21.15.63:42034) 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3ca800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f07b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 -- 172.21.15.63:0/1180933006 learned_addr learned my addr 172.21.15.63:0/1180933006 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3ca800640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 -- 172.21.15.63:0/1180933006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5eff9b0 msgr2=0x55b8e5f07b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f07b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 -- 172.21.15.63:0/1180933006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 msgr2=0x55b8e5f085b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f085b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 -- 172.21.15.63:0/1180933006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e5c44620 con 0x55b8e5f002d0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3ca800640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f085b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3c9400640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe3c406a010 tx=0x7fe3b40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3589ad98edcc41d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3a3400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.033+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4035040 con 0x55b8e5f002d0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe3b4005450 con 0x55b8e5f002d0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3c00054d0 con 0x55b8e5f002d0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3b40044e0 con 0x55b8e5f002d0 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 10 client.25697 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8e5f19a90 con 0x55b8e5f002d0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3c00054d0 con 0x55b8e5f002d0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe3b4003a30 con 0x55b8e5f002d0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 10 client.25697 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.034+0000 7fe3cf5f1740 20 client.25697 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f1afa0 0x55b8e5f1d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3cf5f1740 4 client.25697 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8e5e798b0 con 0x55b8e5f1afa0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3ca800640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f1afa0 0x55b8e5f1d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3ca800640 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f1afa0 0x55b8e5f1d390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f3dfe10c0ef8c35 server_cookie=9d155099d06a8730 in_seq=0 out_seq=0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.035+0000 7fe3a3400640 10 client.25697 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3a3400640 1 -- 172.21.15.63:0/1180933006 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b8e5e798b0 con 0x55b8e5f1afa0 2024-04-25T18:50:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3a3400640 10 client.25697 handle_command_reply: tid=0 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 client.25697 shutdown 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 msgr2=0x55b8e5f08070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe3c406a010 tx=0x7fe3b40042f0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 shutdown_connections 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f1afa0 0x55b8e5f1d390 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f002d0 0x55b8e5f08070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f085b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/1180933006 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f07b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3a3400640 0 client.25697 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.036+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 >> 172.21.15.63:0/1180933006 conn(0x55b8e5efbec0 msgr2=0x55b8e5f01bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.037+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 shutdown_connections 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.037+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/1180933006 wait complete. 2024-04-25T18:50:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.037+0000 7fe3cf5f1740 20 client.25697 trim_cache size 0 max 16384 2024-04-25T18:50:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 Processor -- start 2024-04-25T18:50:34.056 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 -- start start 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f16680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41b2580 con 0x55b8e5f002d0 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41afdc0 con 0x55b8e5eff9b0 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41adc10 con 0x55b8e5f0ab90 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.055+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55526/0 (socket says 172.21.15.63:55526) 2024-04-25T18:50:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 -- 172.21.15.63:0/2100600788 learned_addr learned my addr 172.21.15.63:0/2100600788 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3ca800640 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f16680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3ca800640 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 -- 172.21.15.63:0/2100600788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f0ab90 msgr2=0x55b8e5f231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f231e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 -- 172.21.15.63:0/2100600788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 msgr2=0x55b8e5f16680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f16680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 -- 172.21.15.63:0/2100600788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8e5efb7a0 con 0x55b8e5f002d0 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.056+0000 7fe3c9400640 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4003e80 tx=0x7fe3b402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8925b99972e38b18 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3ae600640 1 -- 172.21.15.63:0/2100600788 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4000e60 con 0x55b8e5f002d0 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3ae600640 1 -- 172.21.15.63:0/2100600788 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3b4040070 con 0x55b8e5f002d0 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3ae600640 1 -- 172.21.15.63:0/2100600788 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4035030 con 0x55b8e5f002d0 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2100600788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 msgr2=0x55b8e5f22ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4003e80 tx=0x7fe3b402a410 comp rx=0 tx=0).stop 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2100600788 shutdown_connections 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f231e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5f002d0 0x55b8e5f22ca0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/2100600788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f16680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.057+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2100600788 >> 172.21.15.63:0/2100600788 conn(0x55b8e5efbec0 msgr2=0x55b8e5f02e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.058+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2100600788 shutdown_connections 2024-04-25T18:50:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.058+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/2100600788 wait complete. 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.058+0000 7fe3cf5f1740 1 -- start start 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.058+0000 7fe3cf5f1740 1 Processor -- start 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f12dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f106a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41ae280 con 0x55b8e5eff9b0 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41af6f0 con 0x55b8e5f0ab90 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c41ae950 con 0x55b8e5f002d0 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3cf5f1740 10 client.0 _ll_register_callbacks cb 0x7fe3b4009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:37224/0 (socket says 172.21.15.63:37224) 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 -- 172.21.15.63:0/3573042280 learned_addr learned my addr 172.21.15.63:0/3573042280 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3ca800640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f12dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3ca800640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f106a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 -- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f0ab90 msgr2=0x55b8e5f106a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f106a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 -- 172.21.15.63:0/3573042280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5eff9b0 msgr2=0x55b8e5f12dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f12dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.059+0000 7fe3c9400640 1 -- 172.21.15.63:0/3573042280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8e5f10d10 con 0x55b8e5f002d0 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3c9400640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402a9d0 tx=0x7fe3b4005500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb0417b5a753c9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3a3400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:50:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4052070 con 0x55b8e5f002d0 2024-04-25T18:50:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3b4040070 con 0x55b8e5f002d0 2024-04-25T18:50:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe3b4031dd0 con 0x55b8e5f002d0 2024-04-25T18:50:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3b4063050 con 0x55b8e5f002d0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3cf5f1740 10 client.25720 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.060+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8e5f813c0 con 0x55b8e5f002d0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3b4040070 con 0x55b8e5f002d0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe3b4004800 con 0x55b8e5f002d0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 10 client.25720 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 10 client.25720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 20 client.25720 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f819b0 0x55b8e5f83dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 4 client.25720 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8e5e73320 con 0x55b8e5f819b0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.061+0000 7fe3ca800640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f819b0 0x55b8e5f83dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.062+0000 7fe3ca800640 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f819b0 0x55b8e5f83dd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9a9586293472fe server_cookie=20830c8ff54fb4a0 in_seq=0 out_seq=0 2024-04-25T18:50:34.065 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.062+0000 7fe3a3400640 10 client.25720 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:34.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3a3400640 1 -- 172.21.15.63:0/3573042280 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8e5e73320 con 0x55b8e5f819b0 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3a3400640 10 client.25720 handle_command_reply: tid=0 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 client.25720 shutdown 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 msgr2=0x55b8e5f10160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402a9d0 tx=0x7fe3b4005500 comp rx=0 tx=0).stop 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 shutdown_connections 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b8e5f819b0 0x55b8e5f83dd0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b8e5f0ab90 0x55b8e5f106a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b8e5f002d0 0x55b8e5f10160 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 --2- 172.21.15.63:0/3573042280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b8e5eff9b0 0x55b8e5f12dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3a3400640 0 client.25720 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 >> 172.21.15.63:0/3573042280 conn(0x55b8e5efbec0 msgr2=0x55b8e5efa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.108+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 shutdown_connections 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cf5f1740 1 -- 172.21.15.63:0/3573042280 wait complete. 2024-04-25T18:50:34.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cf5f1740 20 client.25720 trim_cache size 0 max 16384 2024-04-25T18:50:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:50:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:50:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:50:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 msgr2=0x7fe3b00a6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 0x7fe3b00a6200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc099240 tx=0x7fe3bc0d48b0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 msgr2=0x7fe3c4102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.109+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00ebc0 tx=0x7fe3bc00c6a0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 shutdown_connections 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3b00a3d40 0x7fe3b00a6200 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3c41033e0 0x7fe3c41a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe3c4105fa0 0x7fe3c4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 --2- 172.21.15.63:0/1265033330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe3c4100ed0 0x7fe3c4102960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 >> 172.21.15.63:0/1265033330 conn(0x7fe3c40fc770 msgr2=0x7fe3c406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:34.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 shutdown_connections 2024-04-25T18:50:34.113 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:34.110+0000 7fe3cb200640 1 -- 172.21.15.63:0/1265033330 wait complete. 2024-04-25T18:50:34.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 108904 7.12 MB/sec execute 33 sec latency 45.003 ms 2024-04-25T18:50:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:34 smithi114 ceph-mon[32988]: pgmap v124: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:34.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:34 smithi063 ceph-mon[28006]: pgmap v124: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:34.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:34 smithi070 ceph-mon[32820]: pgmap v124: 97 pgs: 97 active+clean; 48 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-04-25T18:50:35.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 110294 7.25 MB/sec execute 34 sec latency 19.164 ms 2024-04-25T18:50:36.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 111577 7.20 MB/sec execute 35 sec latency 19.797 ms 2024-04-25T18:50:36.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:36 smithi114 ceph-mon[32988]: pgmap v125: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-04-25T18:50:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:36 smithi063 ceph-mon[28006]: pgmap v125: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-04-25T18:50:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:36 smithi070 ceph-mon[32820]: pgmap v125: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-04-25T18:50:37.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 113389 7.26 MB/sec execute 36 sec latency 12.013 ms 2024-04-25T18:50:38.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 114427 7.21 MB/sec execute 37 sec latency 62.651 ms 2024-04-25T18:50:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:38 smithi114 ceph-mon[32988]: pgmap v126: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T18:50:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:38 smithi063 ceph-mon[28006]: pgmap v126: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T18:50:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:38 smithi070 ceph-mon[32820]: pgmap v126: 97 pgs: 97 active+clean; 51 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 161 op/s 2024-04-25T18:50:39.449 INFO:tasks.workunit.client.0.smithi063.stdout: 1 116152 7.21 MB/sec execute 38 sec latency 14.679 ms 2024-04-25T18:50:40.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 117562 7.31 MB/sec execute 39 sec latency 13.381 ms 2024-04-25T18:50:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:40 smithi114 ceph-mon[32988]: pgmap v127: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2024-04-25T18:50:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:40 smithi063 ceph-mon[28006]: pgmap v127: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2024-04-25T18:50:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:40 smithi070 ceph-mon[32820]: pgmap v127: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2024-04-25T18:50:41.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 118951 7.26 MB/sec execute 40 sec latency 18.177 ms 2024-04-25T18:50:42.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 120742 7.39 MB/sec execute 41 sec latency 13.676 ms 2024-04-25T18:50:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:42 smithi114 ceph-mon[32988]: pgmap v128: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:42 smithi063 ceph-mon[28006]: pgmap v128: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:42 smithi070 ceph-mon[32820]: pgmap v128: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:43.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 121612 7.26 MB/sec execute 42 sec latency 44.775 ms 2024-04-25T18:50:44.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 123208 7.26 MB/sec execute 43 sec latency 22.555 ms 2024-04-25T18:50:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:44 smithi114 ceph-mon[32988]: pgmap v129: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:44 smithi063 ceph-mon[28006]: pgmap v129: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:44 smithi070 ceph-mon[32820]: pgmap v129: 97 pgs: 97 active+clean; 51 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-04-25T18:50:45.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 124489 7.34 MB/sec execute 44 sec latency 20.710 ms 2024-04-25T18:50:46.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 125654 7.30 MB/sec execute 45 sec latency 19.611 ms 2024-04-25T18:50:46.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:46 smithi114 ceph-mon[32988]: pgmap v130: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2024-04-25T18:50:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:46 smithi063 ceph-mon[28006]: pgmap v130: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2024-04-25T18:50:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:46 smithi070 ceph-mon[32820]: pgmap v130: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2024-04-25T18:50:47.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 -- 172.21.15.63:0/1991090730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0074b70 msgr2=0x7f79a010e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 --2- 172.21.15.63:0/1991090730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0074b70 0x7f79a010e300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f798c009900 tx=0x7f798c02a480 comp rx=0 tx=0).stop 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 -- 172.21.15.63:0/1991090730 shutdown_connections 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 --2- 172.21.15.63:0/1991090730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a010e840 0x7f79a0110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 --2- 172.21.15.63:0/1991090730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a0074b70 0x7f79a010e300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 --2- 172.21.15.63:0/1991090730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a0074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 -- 172.21.15.63:0/1991090730 >> 172.21.15.63:0/1991090730 conn(0x7f79a00793e0 msgr2=0x7f79a00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 -- 172.21.15.63:0/1991090730 shutdown_connections 2024-04-25T18:50:47.252 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.249+0000 7f79a7200640 1 -- 172.21.15.63:0/1991090730 wait complete. 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.250+0000 7f79a7200640 1 Processor -- start 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.250+0000 7f79a7200640 1 -- start start 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a0074b70 0x7f79a01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a010e840 0x7f79a019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79a0114440 con 0x7f79a0074b70 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01142c0 con 0x7f79a0074250 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a7200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01145c0 con 0x7f79a010e840 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a0074b70 0x7f79a01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f79a5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a010e840 0x7f79a019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38534/0 (socket says 172.21.15.63:38534) 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 -- 172.21.15.63:0/2731506131 learned_addr learned my addr 172.21.15.63:0/2731506131 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 -- 172.21.15.63:0/2731506131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a010e840 msgr2=0x7f79a019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a010e840 0x7f79a019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 -- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a0074b70 msgr2=0x7f79a01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a0074b70 0x7f79a01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.251+0000 7f799fe00640 1 -- 172.21.15.63:0/2731506131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a019af20 con 0x7f79a0074250 2024-04-25T18:50:47.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f799fe00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f799000e9f0 tx=0x7f799000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f799cc00640 1 -- 172.21.15.63:0/2731506131 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799000cd80 con 0x7f79a0074250 2024-04-25T18:50:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f799cc00640 1 -- 172.21.15.63:0/2731506131 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79900054d0 con 0x7f79a0074250 2024-04-25T18:50:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a019b210 con 0x7f79a0074250 2024-04-25T18:50:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f799cc00640 1 -- 172.21.15.63:0/2731506131 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990010690 con 0x7f79a0074250 2024-04-25T18:50:47.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.252+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a01a7520 con 0x7f79a0074250 2024-04-25T18:50:47.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.256+0000 7f79aa1ec740 1 Processor -- start 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.256+0000 7f79aa1ec740 1 -- start start 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.256+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea91295c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.256+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea912a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.256+0000 7f799cc00640 1 -- 172.21.15.63:0/2731506131 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7990004e90 con 0x7f79a0074250 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ea911fb900 con 0x55ea912a0a00 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ea911a1ef0 con 0x55ea91295820 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ea911f78c0 con 0x55ea91296140 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f799fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea912a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea91295c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f799f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f799fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea912a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33774/0 (socket says 172.21.15.63:33774) 2024-04-25T18:50:47.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f799fe00640 1 -- 172.21.15.63:0/649209666 learned_addr learned my addr 172.21.15.63:0/649209666 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 -- 172.21.15.63:0/649209666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 msgr2=0x55ea912a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea912a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 -- 172.21.15.63:0/649209666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 msgr2=0x55ea912a2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f79a5200640 1 -- 172.21.15.63:0/649209666 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea90f0e210 con 0x55ea91295820 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.257+0000 7f799f400640 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.258+0000 7f79a5200640 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea91295c00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f79a01a1230 tx=0x7f7998011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fb7cfd73a7ec393 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.258+0000 7f7995600640 1 -- 172.21.15.63:0/649209666 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998004d20 con 0x55ea91295820 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.258+0000 7f7995600640 1 -- 172.21.15.63:0/649209666 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7998004050 con 0x55ea91295820 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.258+0000 7f7995600640 1 -- 172.21.15.63:0/649209666 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7998005280 con 0x55ea91295820 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.258+0000 7f799cc00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 0x7f79840a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f799cc00640 1 -- 172.21.15.63:0/2731506131 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f79900d0c70 con 0x7f79a0074250 2024-04-25T18:50:47.262 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f799fe00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 0x7f79840a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 -- 172.21.15.63:0/649209666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 msgr2=0x55ea91295c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea91295c00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f79a01a1230 tx=0x7f7998011810 comp rx=0 tx=0).stop 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 -- 172.21.15.63:0/649209666 shutdown_connections 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea912a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/649209666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea91295c00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f799fe00640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 0x7f79840a6280 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79900ce3b0 tx=0x7f79900d4720 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.259+0000 7f79aa1ec740 1 -- 172.21.15.63:0/649209666 >> 172.21.15.63:0/649209666 conn(0x55ea91291d30 msgr2=0x55ea91292110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.260+0000 7f79aa1ec740 1 -- 172.21.15.63:0/649209666 shutdown_connections 2024-04-25T18:50:47.263 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.260+0000 7f79aa1ec740 1 -- 172.21.15.63:0/649209666 wait complete. 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.260+0000 7f79aa1ec740 1 -- start start 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 Processor -- start 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea9129da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea9129e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ea912a3630 con 0x55ea912a0a00 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ea911b5310 con 0x55ea91295820 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ea912a37b0 con 0x55ea91296140 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79aa1ec740 10 client.0 _ll_register_callbacks cb 0x55ea91183cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f799f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.266 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f79a5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea9129da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.261+0000 7f799f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33778/0 (socket says 172.21.15.63:33778) 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 -- 172.21.15.63:0/3255542425 learned_addr learned my addr 172.21.15.63:0/3255542425 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f79a5200640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea9129e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 -- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 msgr2=0x55ea9129da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea9129da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 -- 172.21.15.63:0/3255542425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 msgr2=0x55ea9129e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea9129e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 -- 172.21.15.63:0/3255542425 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea90fda620 con 0x55ea91296140 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f799f400640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f798c006010 tx=0x7f798c004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=761aae34cadd3bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.262+0000 7f798a000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035040 con 0x55ea91296140 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f798c005450 con 0x55ea91296140 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7998004050 con 0x55ea91296140 2024-04-25T18:50:47.267 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f798c005650 con 0x55ea91296140 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f79aa1ec740 10 client.25732 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea912af920 con 0x55ea91296140 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.263+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7998004050 con 0x55ea91296140 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f798c003920 con 0x55ea91296140 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 10 client.25732 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 10 client.25732 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 20 client.25732 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea912b0e80 0x55ea912b3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 4 client.25732 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:50:47.268 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea9120f8b0 con 0x55ea912b0e80 2024-04-25T18:50:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79a5200640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea912b0e80 0x55ea912b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f79a5200640 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea912b0e80 0x55ea912b3290 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9be2e57e2b42f56 server_cookie=5f632d2952c8b2f0 in_seq=0 out_seq=0 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.264+0000 7f798a000640 10 client.25732 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.265+0000 7f798a000640 1 -- 172.21.15.63:0/3255542425 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea9120f8b0 con 0x55ea912b0e80 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.265+0000 7f798a000640 10 client.25732 handle_command_reply: tid=0 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.265+0000 7f79aa1ec740 1 client.25732 shutdown 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 msgr2=0x55ea9129df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f798c006010 tx=0x7f798c004360 comp rx=0 tx=0).stop 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 shutdown_connections 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea912b0e80 0x55ea912b3290 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea9129e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91296140 0x55ea9129df60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f798a000640 0 client.25732 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/3255542425 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91295820 0x55ea9129da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 >> 172.21.15.63:0/3255542425 conn(0x55ea91291d30 msgr2=0x55ea91297a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 shutdown_connections 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 1 -- 172.21.15.63:0/3255542425 wait complete. 2024-04-25T18:50:47.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.266+0000 7f79aa1ec740 20 client.25732 trim_cache size 0 max 16384 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 Processor -- start 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 -- start start 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea91295820 0x55ea91291610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea912a0a00 0x55ea912b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01b2740 con 0x55ea91295820 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01aff80 con 0x55ea91296140 2024-04-25T18:50:47.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.285+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01addd0 con 0x55ea912a0a00 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38546/0 (socket says 172.21.15.63:38546) 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 -- 172.21.15.63:0/4164622575 learned_addr learned my addr 172.21.15.63:0/4164622575 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f79a5200640 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea91295820 0x55ea91291610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f79a5200640 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea912a0a00 0x55ea912b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 -- 172.21.15.63:0/4164622575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea912a0a00 msgr2=0x55ea912b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea912a0a00 0x55ea912b8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 -- 172.21.15.63:0/4164622575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea91295820 msgr2=0x55ea91291610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea91295820 0x55ea91291610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 -- 172.21.15.63:0/4164622575 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea9129c9f0 con 0x55ea91296140 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.286+0000 7f799f400640 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f798c005ab0 tx=0x7f798c031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5c8790e021335e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.287+0000 7f7994c00640 1 -- 172.21.15.63:0/4164622575 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c051030 con 0x55ea91296140 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.287+0000 7f7994c00640 1 -- 172.21.15.63:0/4164622575 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f798c040070 con 0x55ea91296140 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.287+0000 7f7994c00640 1 -- 172.21.15.63:0/4164622575 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035070 con 0x55ea91296140 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.287+0000 7f79aa1ec740 1 -- 172.21.15.63:0/4164622575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 msgr2=0x55ea9129e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.287+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f798c005ab0 tx=0x7f798c031400 comp rx=0 tx=0).stop 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 -- 172.21.15.63:0/4164622575 shutdown_connections 2024-04-25T18:50:47.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea9129e2c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea912a0a00 0x55ea912b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/4164622575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea91295820 0x55ea91291610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 -- 172.21.15.63:0/4164622575 >> 172.21.15.63:0/4164622575 conn(0x55ea91291d30 msgr2=0x55ea91297a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 -- 172.21.15.63:0/4164622575 shutdown_connections 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 -- 172.21.15.63:0/4164622575 wait complete. 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 -- start start 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.288+0000 7f79aa1ec740 1 Processor -- start 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91295820 0x55ea912a6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01ae440 con 0x55ea912a0a00 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01af8b0 con 0x55ea91296140 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79a01aeb10 con 0x55ea91295820 2024-04-25T18:50:47.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79aa1ec740 10 client.0 _ll_register_callbacks cb 0x7f798c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38560/0 (socket says 172.21.15.63:38560) 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 -- 172.21.15.63:0/1199887932 learned_addr learned my addr 172.21.15.63:0/1199887932 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79a5200640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91295820 0x55ea912a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f79a5200640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 -- 172.21.15.63:0/1199887932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91295820 msgr2=0x55ea912a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91295820 0x55ea912a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 -- 172.21.15.63:0/1199887932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 msgr2=0x55ea912a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.289+0000 7f799f400640 1 -- 172.21.15.63:0/1199887932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea91316630 con 0x55ea91296140 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f799f400640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f798c02a9a0 tx=0x7f798c003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8168e50415b1eca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f798a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798c035070 con 0x55ea91296140 2024-04-25T18:50:47.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f798c0319d0 con 0x55ea91296140 2024-04-25T18:50:47.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f798c040070 con 0x55ea91296140 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f798c062050 con 0x55ea91296140 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f79aa1ec740 10 client.25721 fetch_fsmap learned FSMap version 28 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.290+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ea91317900 con 0x55ea91296140 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f798c040070 con 0x55ea91296140 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7f798c05d030 con 0x55ea91296140 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 10 client.25721 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 10 client.25721 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 20 client.25721 populate_metadata read hostname 'smithi063' 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea91317fc0 0x55ea9131a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 4 client.25721 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ea91209320 con 0x55ea91317fc0 2024-04-25T18:50:47.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.291+0000 7f79a5200640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea91317fc0 0x55ea9131a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:50:47.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.292+0000 7f79a5200640 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea91317fc0 0x55ea9131a3e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=536ef69d66473e35 server_cookie=1f9646590770846d in_seq=0 out_seq=0 2024-04-25T18:50:47.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.292+0000 7f798a000640 10 client.25721 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f798a000640 1 -- 172.21.15.63:0/1199887932 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ea91209320 con 0x55ea91317fc0 2024-04-25T18:50:47.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f798a000640 10 client.25721 handle_command_reply: tid=0 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 client.25721 shutdown 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 msgr2=0x55ea912a6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f798c02a9a0 tx=0x7f798c003c00 comp rx=0 tx=0).stop 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 shutdown_connections 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ea91317fc0 0x55ea9131a3e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ea91296140 0x55ea912a6ba0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea912a0a00 0x55ea912a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 --2- 172.21.15.63:0/1199887932 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ea91295820 0x55ea912a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f798a000640 0 client.25721 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 >> 172.21.15.63:0/1199887932 conn(0x55ea91291d30 msgr2=0x55ea9129f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 shutdown_connections 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 1 -- 172.21.15.63:0/1199887932 wait complete. 2024-04-25T18:50:47.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.348+0000 7f79aa1ec740 20 client.25721 trim_cache size 0 max 16384 2024-04-25T18:50:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:50:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:50:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:50:47.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 msgr2=0x7f79840a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 0x7f79840a6280 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79900ce3b0 tx=0x7f79900d4720 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 msgr2=0x7f79a01a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f799000e9f0 tx=0x7f799000edf0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 shutdown_connections 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79840a3dc0 0x7f79840a6280 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79a010e840 0x7f79a019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79a0074b70 0x7f79a01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 --2- 172.21.15.63:0/2731506131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79a0074250 0x7f79a01a05a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 >> 172.21.15.63:0/2731506131 conn(0x7f79a00793e0 msgr2=0x7f79a00fee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 shutdown_connections 2024-04-25T18:50:47.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:50:47.349+0000 7f79a7200640 1 -- 172.21.15.63:0/2731506131 wait complete. 2024-04-25T18:50:47.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 127195 7.30 MB/sec execute 46 sec latency 55.235 ms 2024-04-25T18:50:47.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:47 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:47.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:47 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:47 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:50:48.450 INFO:tasks.workunit.client.0.smithi063.stdout: 1 128328 7.29 MB/sec execute 47 sec latency 27.378 ms 2024-04-25T18:50:48.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:48 smithi114 ceph-mon[32988]: pgmap v131: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:48 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:48 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:48 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:50:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:48 smithi063 ceph-mon[28006]: pgmap v131: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T18:50:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:48 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:48 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:48.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:48 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:50:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:48 smithi070 ceph-mon[32820]: pgmap v131: 97 pgs: 97 active+clean; 55 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 203 op/s 2024-04-25T18:50:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:48 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:50:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:48 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:50:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:48 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:50:49.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 129887 7.25 MB/sec execute 48 sec latency 16.218 ms 2024-04-25T18:50:50.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 131352 7.35 MB/sec execute 49 sec latency 14.602 ms 2024-04-25T18:50:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:50 smithi114 ceph-mon[32988]: pgmap v132: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-04-25T18:50:50.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:50 smithi063 ceph-mon[28006]: pgmap v132: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-04-25T18:50:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:50 smithi070 ceph-mon[32820]: pgmap v132: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 322 op/s 2024-04-25T18:50:51.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 132190 7.25 MB/sec execute 50 sec latency 19.693 ms 2024-04-25T18:50:52.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 133773 7.24 MB/sec execute 51 sec latency 14.522 ms 2024-04-25T18:50:52.755 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:52 smithi070 ceph-mon[32820]: pgmap v133: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:52.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:52 smithi114 ceph-mon[32988]: pgmap v133: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:52.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:52 smithi063 ceph-mon[28006]: pgmap v133: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:53.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 135097 7.31 MB/sec execute 52 sec latency 50.379 ms 2024-04-25T18:50:54.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 136041 7.22 MB/sec execute 53 sec latency 20.487 ms 2024-04-25T18:50:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:54 smithi114 ceph-mon[32988]: pgmap v134: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:54.880 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:54 smithi063 ceph-mon[28006]: pgmap v134: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:54 smithi070 ceph-mon[32820]: pgmap v134: 97 pgs: 97 active+clean; 50 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2024-04-25T18:50:55.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 137594 7.25 MB/sec execute 54 sec latency 16.877 ms 2024-04-25T18:50:56.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 138753 7.27 MB/sec execute 55 sec latency 15.645 ms 2024-04-25T18:50:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:56 smithi114 ceph-mon[32988]: pgmap v135: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 277 op/s 2024-04-25T18:50:56.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:56 smithi063 ceph-mon[28006]: pgmap v135: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 277 op/s 2024-04-25T18:50:56.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:56 smithi070 ceph-mon[32820]: pgmap v135: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 277 op/s 2024-04-25T18:50:57.451 INFO:tasks.workunit.client.0.smithi063.stdout: 1 139873 7.23 MB/sec execute 56 sec latency 22.537 ms 2024-04-25T18:50:58.452 INFO:tasks.workunit.client.0.smithi063.stdout: 1 141378 7.23 MB/sec execute 57 sec latency 60.821 ms 2024-04-25T18:50:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:50:58 smithi114 ceph-mon[32988]: pgmap v136: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-25T18:50:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:50:58 smithi063 ceph-mon[28006]: pgmap v136: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-25T18:50:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:50:58 smithi070 ceph-mon[32820]: pgmap v136: 97 pgs: 97 active+clean; 53 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 194 op/s 2024-04-25T18:50:59.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 142465 7.22 MB/sec execute 58 sec latency 17.755 ms 2024-04-25T18:51:00.452 INFO:tasks.workunit.client.0.smithi063.stdout: 1 143687 7.19 MB/sec execute 59 sec latency 20.028 ms 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.501+0000 7f8b2b800640 1 -- 172.21.15.63:0/696416506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b241043b0 msgr2=0x7f8b24104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.501+0000 7f8b2b800640 1 --2- 172.21.15.63:0/696416506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b241043b0 0x7f8b24104790 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009860 tx=0x7f8b1802a340 comp rx=0 tx=0).stop 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 -- 172.21.15.63:0/696416506 shutdown_connections 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 --2- 172.21.15.63:0/696416506 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24104d60 0x7f8b24108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 --2- 172.21.15.63:0/696416506 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b24109480 0x7f8b24111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 --2- 172.21.15.63:0/696416506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b241043b0 0x7f8b24104790 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 -- 172.21.15.63:0/696416506 >> 172.21.15.63:0/696416506 conn(0x7f8b240ffc50 msgr2=0x7f8b24102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 -- 172.21.15.63:0/696416506 shutdown_connections 2024-04-25T18:51:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.502+0000 7f8b2b800640 1 -- 172.21.15.63:0/696416506 wait complete. 2024-04-25T18:51:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.503+0000 7f8b2b800640 1 Processor -- start 2024-04-25T18:51:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.503+0000 7f8b2b800640 1 -- start start 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.503+0000 7f8b2b800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b2b800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b24109480 0x7f8b241a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b2b800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 0x7f8b2419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b2b800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241143f0 con 0x7f8b24109480 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b2b800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b24114270 con 0x7f8b241043b0 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b2b800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8b24114570 con 0x7f8b2419a750 2024-04-25T18:51:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b29800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 0x7f8b2419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58834/0 (socket says 172.21.15.63:58834) 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 -- 172.21.15.63:0/1856365104 learned_addr learned my addr 172.21.15.63:0/1856365104 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b29800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 0x7f8b2419abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50366/0 (socket says 172.21.15.63:50366) 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b23e00640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b24109480 0x7f8b241a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 -- 172.21.15.63:0/1856365104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 msgr2=0x7f8b2419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 0x7f8b2419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 -- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b24109480 msgr2=0x7f8b241a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b24109480 0x7f8b241a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.504+0000 7f8b28e00640 1 -- 172.21.15.63:0/1856365104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2419b350 con 0x7f8b241043b0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b18035040 tx=0x7f8b18032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b21600640 1 -- 172.21.15.63:0/1856365104 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18037040 con 0x7f8b241043b0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b21600640 1 -- 172.21.15.63:0/1856365104 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b18041070 con 0x7f8b241043b0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2419b5e0 con 0x7f8b241043b0 2024-04-25T18:51:00.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b21600640 1 -- 172.21.15.63:0/1856365104 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18003850 con 0x7f8b241043b0 2024-04-25T18:51:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.505+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b241a7710 con 0x7f8b241043b0 2024-04-25T18:51:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 Processor -- start 2024-04-25T18:51:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b21600640 1 -- 172.21.15.63:0/1856365104 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8b180039f0 con 0x7f8b241043b0 2024-04-25T18:51:00.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 -- start start 2024-04-25T18:51:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7a820 0x5631ccb7ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb87df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5631ccae0900 con 0x5631ccb7a820 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5631cca86ef0 con 0x5631ccb85a00 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.510+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631ccadc8c0 con 0x5631ccb7b140 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b29800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7a820 0x5631ccb7ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b23e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb87df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50376/0 (socket says 172.21.15.63:50376) 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 -- 172.21.15.63:0/1619165037 learned_addr learned my addr 172.21.15.63:0/1619165037 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 -- 172.21.15.63:0/1619165037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 msgr2=0x5631ccb87df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb87df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 -- 172.21.15.63:0/1619165037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7a820 msgr2=0x5631ccb7ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7a820 0x5631ccb7ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 -- 172.21.15.63:0/1619165037 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631cc7f3210 con 0x5631ccb7b140 2024-04-25T18:51:00.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.511+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8b180edbf0 tx=0x7f8b180f6510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8004550bb8c7c6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b16000640 1 -- 172.21.15.63:0/1619165037 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180f0a80 con 0x5631ccb7b140 2024-04-25T18:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b21600640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 0x7f8b0c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b16000640 1 -- 172.21.15.63:0/1619165037 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b180fd030 con 0x5631ccb7b140 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b16000640 1 -- 172.21.15.63:0/1619165037 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18102750 con 0x5631ccb7b140 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b21600640 1 -- 172.21.15.63:0/1856365104 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f8b180f0490 con 0x7f8b241043b0 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.512+0000 7f8b29800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 0x7f8b0c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1619165037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 msgr2=0x5631ccb854c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8b180edbf0 tx=0x7f8b180f6510 comp rx=0 tx=0).stop 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1619165037 shutdown_connections 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb87df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7b140 0x5631ccb854c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1619165037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7a820 0x5631ccb7ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1619165037 >> 172.21.15.63:0/1619165037 conn(0x5631ccb76d30 msgr2=0x5631ccb77110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1619165037 shutdown_connections 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b29800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 0x7f8b0c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c005fd0 tx=0x7f8b1c009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1619165037 wait complete. 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.513+0000 7f8b2e7ca740 1 -- start start 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 Processor -- start 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb82ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb833e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5631ccb883f0 con 0x5631ccb7b140 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5631cca9a310 con 0x5631ccb85a00 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5631ccb88570 con 0x5631ccb7a820 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b2e7ca740 10 client.0 _ll_register_callbacks cb 0x5631cca68cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50392/0 (socket says 172.21.15.63:50392) 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b28e00640 1 -- 172.21.15.63:0/1484536732 learned_addr learned my addr 172.21.15.63:0/1484536732 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b23e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb82ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.514+0000 7f8b29800640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb833e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 -- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 msgr2=0x5631ccb833e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb833e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 -- 172.21.15.63:0/1484536732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 msgr2=0x5631ccb82ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb82ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 -- 172.21.15.63:0/1484536732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631cc8bf620 con 0x5631ccb7a820 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b28e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b180b8ea0 tx=0x7f8b18030630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b23307519f819609 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180b8770 con 0x5631ccb7a820 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f8b18003ce0 con 0x5631ccb7a820 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b180fd030 con 0x5631ccb7a820 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b180bbe30 con 0x5631ccb7a820 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b2e7ca740 10 client.25750 fetch_fsmap learned FSMap version 29 2024-04-25T18:51:00.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.515+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5631ccb883f0 con 0x5631ccb7a820 2024-04-25T18:51:00.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b180fd030 con 0x5631ccb7a820 2024-04-25T18:51:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8b180b8b40 con 0x5631ccb7a820 2024-04-25T18:51:00.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 10 client.25750 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 10 client.25750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 20 client.25750 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccb957b0 0x5631ccb97be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 4 client.25750 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631ccaf48b0 con 0x5631ccb957b0 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.516+0000 7f8b23e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccb957b0 0x5631ccb97be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.517+0000 7f8b23e00640 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccb957b0 0x5631ccb97be0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7164795931255b48 server_cookie=e6a829dcba0badf8 in_seq=0 out_seq=0 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.517+0000 7f8b0aa00640 10 client.25750 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.517+0000 7f8b0aa00640 1 -- 172.21.15.63:0/1484536732 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631ccaf48b0 con 0x5631ccb957b0 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.517+0000 7f8b0aa00640 10 client.25750 handle_command_reply: tid=0 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 client.25750 shutdown 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 msgr2=0x5631ccb82960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8b180b8ea0 tx=0x7f8b18030630 comp rx=0 tx=0).stop 2024-04-25T18:51:00.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 shutdown_connections 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccb957b0 0x5631ccb97be0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb833e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb82ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/1484536732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb82960 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 >> 172.21.15.63:0/1484536732 conn(0x5631ccb76d30 msgr2=0x5631ccb7cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b0aa00640 0 client.25750 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 shutdown_connections 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/1484536732 wait complete. 2024-04-25T18:51:00.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.518+0000 7f8b2e7ca740 20 client.25750 trim_cache size 0 max 16384 2024-04-25T18:51:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.540+0000 7f8b2e7ca740 1 Processor -- start 2024-04-25T18:51:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.540+0000 7f8b2e7ca740 1 -- start start 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.540+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb75d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 0x5631ccb9d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241ae230 con 0x5631ccb7b140 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241abaa0 con 0x5631ccb7a820 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241a9910 con 0x5631ccb85a00 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b29800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 0x5631ccb9d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b29800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 0x5631ccb9d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:50404/0 (socket says 172.21.15.63:50404) 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58868/0 (socket says 172.21.15.63:58868) 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b29800640 1 -- 172.21.15.63:0/25259993 learned_addr learned my addr 172.21.15.63:0/25259993 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:00.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b23e00640 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb75d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 -- 172.21.15.63:0/25259993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 msgr2=0x5631ccb9d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 0x5631ccb9d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 -- 172.21.15.63:0/25259993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 msgr2=0x5631ccb75d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb75d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.541+0000 7f8b28e00640 1 -- 172.21.15.63:0/25259993 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ccb80ed0 con 0x5631ccb7a820 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.542+0000 7f8b28e00640 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b180b8930 tx=0x7f8b180303b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b57ee74e672311d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.542+0000 7f8b15600640 1 -- 172.21.15.63:0/25259993 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180bc010 con 0x5631ccb7a820 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.542+0000 7f8b15600640 1 -- 172.21.15.63:0/25259993 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b180fd070 con 0x5631ccb7a820 2024-04-25T18:51:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.542+0000 7f8b15600640 1 -- 172.21.15.63:0/25259993 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180bc2a0 con 0x5631ccb7a820 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/25259993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 msgr2=0x5631ccb91220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8b180b8930 tx=0x7f8b180303b0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/25259993 shutdown_connections 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb85a00 0x5631ccb9d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb75d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/25259993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb7a820 0x5631ccb91220 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/25259993 >> 172.21.15.63:0/25259993 conn(0x5631ccb76d30 msgr2=0x5631ccb75600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/25259993 shutdown_connections 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/25259993 wait complete. 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.543+0000 7f8b2e7ca740 1 -- start start 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 Processor -- start 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb8dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb8b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241a9f60 con 0x5631ccb7b140 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241ab3d0 con 0x5631ccb85a00 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241aa630 con 0x5631ccb7a820 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b2e7ca740 10 client.0 _ll_register_callbacks cb 0x7f8b180f62d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b28e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb8dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b29800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b23e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb8b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b29800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58884/0 (socket says 172.21.15.63:58884) 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.544+0000 7f8b29800640 1 -- 172.21.15.63:0/3373556740 learned_addr learned my addr 172.21.15.63:0/3373556740 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 -- 172.21.15.63:0/3373556740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 msgr2=0x5631ccb8dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb8dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 -- 172.21.15.63:0/3373556740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 msgr2=0x5631ccb8b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:00.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb8b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 -- 172.21.15.63:0/3373556740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631ccbfadc0 con 0x5631ccb85a00 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b29800640 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c004260 tx=0x7f8b1c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82028730dfeb366b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b0aa00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c02e060 con 0x5631ccb85a00 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f8b1c00f690 con 0x5631ccb85a00 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.545+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b180fd070 con 0x5631ccb85a00 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8b1c00f8d0 con 0x5631ccb85a00 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 10 client.25739 fetch_fsmap learned FSMap version 29 2024-04-25T18:51:00.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5631ccbfbfd0 con 0x5631ccb85a00 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b180fd070 con 0x5631ccb85a00 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8b1c028070 con 0x5631ccb85a00 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 10 client.25739 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 10 client.25739 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 20 client.25739 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:00.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccbfc730 0x5631ccbfeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 4 client.25739 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:51:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.546+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631ccaee320 con 0x5631ccbfc730 2024-04-25T18:51:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.547+0000 7f8b28e00640 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccbfc730 0x5631ccbfeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.547+0000 7f8b0aa00640 10 client.25739 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:00.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.547+0000 7f8b28e00640 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccbfc730 0x5631ccbfeb50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41164fba3b408320 server_cookie=4d62989d529a0c01 in_seq=0 out_seq=0 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b0aa00640 1 -- 172.21.15.63:0/3373556740 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631ccaee320 con 0x5631ccbfc730 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b0aa00640 10 client.25739 handle_command_reply: tid=0 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 client.25739 shutdown 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 msgr2=0x5631ccb8b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c004260 tx=0x7f8b1c004ab0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.598 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 shutdown_connections 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5631ccbfc730 0x5631ccbfeb50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5631ccb85a00 0x5631ccb8b710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5631ccb7b140 0x5631ccb8b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 --2- 172.21.15.63:0/3373556740 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5631ccb7a820 0x5631ccb8dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b0aa00640 0 client.25739 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.595+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 >> 172.21.15.63:0/3373556740 conn(0x5631ccb76d30 msgr2=0x5631ccb75e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 shutdown_connections 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2e7ca740 1 -- 172.21.15.63:0/3373556740 wait complete. 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2e7ca740 20 client.25739 trim_cache size 0 max 16384 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:51:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:51:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:51:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:51:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 msgr2=0x7f8b0c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 0x7f8b0c0a6300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c005fd0 tx=0x7f8b1c009500 comp rx=0 tx=0).stop 2024-04-25T18:51:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 msgr2=0x7f8b241a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.596+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8b18035040 tx=0x7f8b18032d50 comp rx=0 tx=0).stop 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 shutdown_connections 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8b0c0a3e40 0x7f8b0c0a6300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8b2419a750 0x7f8b2419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8b24109480 0x7f8b241a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 --2- 172.21.15.63:0/1856365104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b241043b0 0x7f8b241a0690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 >> 172.21.15.63:0/1856365104 conn(0x7f8b240ffc50 msgr2=0x7f8b24101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 shutdown_connections 2024-04-25T18:51:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:00.597+0000 7f8b2b800640 1 -- 172.21.15.63:0/1856365104 wait complete. 2024-04-25T18:51:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:00 smithi114 ceph-mon[32988]: pgmap v137: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2024-04-25T18:51:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:00 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:00 smithi063 ceph-mon[28006]: pgmap v137: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2024-04-25T18:51:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:00 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:00 smithi070 ceph-mon[32820]: pgmap v137: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 275 op/s 2024-04-25T18:51:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:00 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:01.452 INFO:tasks.workunit.client.0.smithi063.stdout: 1 145142 7.21 MB/sec execute 60 sec latency 39.533 ms 2024-04-25T18:51:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:01 smithi114 ceph-mon[32988]: pgmap v138: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:01.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:01 smithi063 ceph-mon[28006]: pgmap v138: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:01.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:01 smithi070 ceph-mon[32820]: pgmap v138: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:02.452 INFO:tasks.workunit.client.0.smithi063.stdout: 1 145710 7.18 MB/sec execute 61 sec latency 55.894 ms 2024-04-25T18:51:03.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 146528 7.10 MB/sec execute 62 sec latency 75.497 ms 2024-04-25T18:51:04.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:04 smithi063 ceph-mon[28006]: pgmap v139: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:04.452 INFO:tasks.workunit.client.0.smithi063.stdout: 1 148374 7.15 MB/sec execute 63 sec latency 17.571 ms 2024-04-25T18:51:04.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:04 smithi070 ceph-mon[32820]: pgmap v139: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:04.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:04 smithi114 ceph-mon[32988]: pgmap v139: 97 pgs: 97 active+clean; 55 MiB data, 785 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 156 op/s 2024-04-25T18:51:05.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:05 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:05.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 149554 7.14 MB/sec execute 64 sec latency 14.407 ms 2024-04-25T18:51:05.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:05 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:05.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:05 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:06.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:06 smithi063 ceph-mon[28006]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-04-25T18:51:06.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 150956 7.12 MB/sec execute 65 sec latency 19.820 ms 2024-04-25T18:51:06.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:06 smithi070 ceph-mon[32820]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-04-25T18:51:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:06 smithi114 ceph-mon[32988]: pgmap v140: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 303 op/s 2024-04-25T18:51:07.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 152655 7.20 MB/sec execute 66 sec latency 13.722 ms 2024-04-25T18:51:08.077 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T18:51:08.077 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-04-25T18:51:08.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:51:08.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:08 smithi063 ceph-mon[28006]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-25T18:51:08.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 153658 7.14 MB/sec execute 67 sec latency 43.184 ms 2024-04-25T18:51:08.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:08 smithi070 ceph-mon[32820]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-25T18:51:08.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:08 smithi114 ceph-mon[32988]: pgmap v141: 97 pgs: 97 active+clean; 54 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 227 op/s 2024-04-25T18:51:08.574 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:09.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 155571 7.19 MB/sec execute 68 sec latency 15.720 ms 2024-04-25T18:51:10.366 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:10 smithi063 ceph-mon[28006]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-04-25T18:51:10.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 156814 7.18 MB/sec execute 69 sec latency 24.068 ms 2024-04-25T18:51:10.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:10 smithi070 ceph-mon[32820]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-04-25T18:51:10.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:10 smithi114 ceph-mon[32988]: pgmap v142: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-04-25T18:51:11.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.306+0000 7fabff600700 1 -- 172.21.15.63:0/3405741302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf810e3a0 msgr2=0x7fabf8115f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:11.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.306+0000 7fabff600700 1 --2- 172.21.15.63:0/3405741302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf810e3a0 0x7fabf8115f70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fabf000b0d0 tx=0x7fabf000b3e0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 -- 172.21.15.63:0/3405741302 shutdown_connections 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 --2- 172.21.15.63:0/3405741302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf810e3a0 0x7fabf8115f70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 --2- 172.21.15.63:0/3405741302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf8109310 0x7fabf81096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 --2- 172.21.15.63:0/3405741302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109cc0 0x7fabf810dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 -- 172.21.15.63:0/3405741302 >> 172.21.15.63:0/3405741302 conn(0x7fabf8076210 msgr2=0x7fabf8078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 -- 172.21.15.63:0/3405741302 shutdown_connections 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.307+0000 7fabff600700 1 -- 172.21.15.63:0/3405741302 wait complete. 2024-04-25T18:51:11.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.308+0000 7fabff600700 1 Processor -- start 2024-04-25T18:51:11.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 -- start start 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 0x7fabf819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf810e3a0 0x7fabf81a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8119290 con 0x7fabf8109310 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8119110 con 0x7fabf810e3a0 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.309+0000 7fabff600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8119410 con 0x7fabf8109cc0 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43704/0 (socket says 172.21.15.63:43704) 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 -- 172.21.15.63:0/3743875960 learned_addr learned my addr 172.21.15.63:0/3743875960 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabfd600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf810e3a0 0x7fabf81a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:11.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabfcc00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 0x7fabf819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:11.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 -- 172.21.15.63:0/3743875960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf810e3a0 msgr2=0x7fabf81a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:11.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf810e3a0 0x7fabf81a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 -- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 msgr2=0x7fabf819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:11.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 0x7fabf819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 -- 172.21.15.63:0/3743875960 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf81a4710 con 0x7fabf8109cc0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabfcc00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 0x7fabf819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.310+0000 7fabf7e00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e970 tx=0x7fabe400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.311+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe400cbd0 con 0x7fabf8109cc0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.311+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabe400cd70 con 0x7fabf8109cc0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.311+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4005680 con 0x7fabf8109cc0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.312+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf81a4a00 con 0x7fabf8109cc0 2024-04-25T18:51:11.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.312+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf81ac280 con 0x7fabf8109cc0 2024-04-25T18:51:11.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.313+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf8069730 con 0x7fabf8109cc0 2024-04-25T18:51:11.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.315+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fabe4004750 con 0x7fabf8109cc0 2024-04-25T18:51:11.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.316+0000 7fabf5600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 0x7fabe00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:11.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.316+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fabe40d09e0 con 0x7fabf8109cc0 2024-04-25T18:51:11.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.316+0000 7fabfcc00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 0x7fabe00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:11.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.317+0000 7fabfcc00700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 0x7fabe00a62f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fabec009cb0 tx=0x7fabec009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:11.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.323+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fabe409d010 con 0x7fabf8109cc0 2024-04-25T18:51:11.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 158453 7.18 MB/sec execute 70 sec latency 17.718 ms 2024-04-25T18:51:11.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.627+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabf802df80 con 0x7fabf8109cc0 2024-04-25T18:51:11.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.628+0000 7fabf5600700 1 -- 172.21.15.63:0/3743875960 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11869 (secure 0 0 0) 0x7fabe409c910 con 0x7fabf8109cc0 2024-04-25T18:51:11.631 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:51:11.631 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:51:03.994152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 msgr2=0x7fabe00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 0x7fabe00a62f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fabec009cb0 tx=0x7fabec009260 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 msgr2=0x7fabf819fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabe400e970 tx=0x7fabe400ec80 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 shutdown_connections 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fabe00a3e30 0x7fabe00a62f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabf810e3a0 0x7fabf81a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabf8109310 0x7fabf819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 --2- 172.21.15.63:0/3743875960 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fabf8109cc0 0x7fabf819fc40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:11.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.631+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 >> 172.21.15.63:0/3743875960 conn(0x7fabf8076210 msgr2=0x7fabf8078530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:11.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.632+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 shutdown_connections 2024-04-25T18:51:11.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:11.632+0000 7fabff600700 1 -- 172.21.15.63:0/3743875960 wait complete. 2024-04-25T18:51:11.635 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 30 2024-04-25T18:51:12.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:12 smithi114 ceph-mon[32988]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:12.369 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:12 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3743875960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:12 smithi063 ceph-mon[28006]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:12.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:12 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3743875960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:12.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 159796 7.23 MB/sec execute 71 sec latency 46.769 ms 2024-04-25T18:51:12.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:12 smithi070 ceph-mon[32820]: pgmap v143: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:12.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:12 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3743875960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:12.639 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 82.35 --await-for 42.0' 2024-04-25T18:51:12.640 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 82.35 --await-for 42.0 2024-04-25T18:51:13.120 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:13.453 INFO:tasks.workunit.client.0.smithi063.stdout: 1 160886 7.18 MB/sec execute 72 sec latency 24.622 ms 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 -- 172.21.15.63:0/2592008226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074250 msgr2=0x7f57e0074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 --2- 172.21.15.63:0/2592008226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074250 0x7f57e0074630 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009ca0 tx=0x7f57d402a950 comp rx=0 tx=0).stop 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 -- 172.21.15.63:0/2592008226 shutdown_connections 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 --2- 172.21.15.63:0/2592008226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074b70 0x7f57e010e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 --2- 172.21.15.63:0/2592008226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e0110c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 --2- 172.21.15.63:0/2592008226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074250 0x7f57e0074630 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.752+0000 7f57e8400640 1 -- 172.21.15.63:0/2592008226 >> 172.21.15.63:0/2592008226 conn(0x7f57e00793e0 msgr2=0x7f57e00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.753+0000 7f57e8400640 1 -- 172.21.15.63:0/2592008226 shutdown_connections 2024-04-25T18:51:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.753+0000 7f57e8400640 1 -- 172.21.15.63:0/2592008226 wait complete. 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.753+0000 7f57e8400640 1 Processor -- start 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 -- start start 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074250 0x7f57e01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57e0114400 con 0x7f57e010e840 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57e0114280 con 0x7f57e0074250 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e8400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57e0114580 con 0x7f57e0074b70 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e6400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43722/0 (socket says 172.21.15.63:43722) 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e5000640 1 -- 172.21.15.63:0/3637974575 learned_addr learned my addr 172.21.15.63:0/3637974575 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.754+0000 7f57e5a00640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074250 0x7f57e01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 -- 172.21.15.63:0/3637974575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074250 msgr2=0x7f57e01a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074250 0x7f57e01a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 -- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 msgr2=0x7f57e019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 -- 172.21.15.63:0/3637974575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e019af20 con 0x7f57e0074b70 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e6400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e5000640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9f0 tx=0x7f57cc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57de000640 1 -- 172.21.15.63:0/3637974575 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00cd80 con 0x7f57e0074b70 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57de000640 1 -- 172.21.15.63:0/3637974575 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57cc0054d0 con 0x7f57e0074b70 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e019b210 con 0x7f57e0074b70 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57de000640 1 -- 172.21.15.63:0/3637974575 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc010690 con 0x7f57e0074b70 2024-04-25T18:51:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.755+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e01a7520 con 0x7f57e0074b70 2024-04-25T18:51:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 Processor -- start 2024-04-25T18:51:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57de000640 1 -- 172.21.15.63:0/3637974575 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f57cc004e90 con 0x7f57e0074b70 2024-04-25T18:51:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 -- start start 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 0x5650dc1cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1cba00 0x5650dc1cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5650dc126900 con 0x5650dc1c1140 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5650dc0ccef0 con 0x5650dc1c0820 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57eb164740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650dc1228c0 con 0x5650dc1cba00 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.759+0000 7f57e6400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 0x5650dc1cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 0x5650dc1cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54828/0 (socket says 172.21.15.63:54828) 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e5a00640 1 -- 172.21.15.63:0/2664704360 learned_addr learned my addr 172.21.15.63:0/2664704360 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e5000640 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1cba00 0x5650dc1cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 -- 172.21.15.63:0/2664704360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1cba00 msgr2=0x5650dc1cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1cba00 0x5650dc1cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 -- 172.21.15.63:0/2664704360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 msgr2=0x5650dc1cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 0x5650dc1cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 -- 172.21.15.63:0/2664704360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650dbe39210 con 0x5650dc1c0820 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.760+0000 7f57e6400640 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1c0c00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f57e01a1230 tx=0x7f57d8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=841fc31672053b13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57de000640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 0x7f57c80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57d2a00640 1 -- 172.21.15.63:0/2664704360 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d801e070 con 0x5650dc1c0820 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57d2a00640 1 -- 172.21.15.63:0/2664704360 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d8004310 con 0x5650dc1c0820 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57d2a00640 1 -- 172.21.15.63:0/2664704360 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d8019850 con 0x5650dc1c0820 2024-04-25T18:51:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57de000640 1 -- 172.21.15.63:0/3637974575 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f57cc0d0c10 con 0x7f57e0074b70 2024-04-25T18:51:13.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57e5a00640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 0x7f57c80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 -- 172.21.15.63:0/2664704360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 msgr2=0x5650dc1c0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1c0c00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f57e01a1230 tx=0x7f57d8011810 comp rx=0 tx=0).stop 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 -- 172.21.15.63:0/2664704360 shutdown_connections 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1cba00 0x5650dc1cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c1140 0x5650dc1cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 --2- 172.21.15.63:0/2664704360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1c0c00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.761+0000 7f57eb164740 1 -- 172.21.15.63:0/2664704360 >> 172.21.15.63:0/2664704360 conn(0x5650dc1bcd30 msgr2=0x5650dc1bd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.762+0000 7f57eb164740 1 -- 172.21.15.63:0/2664704360 shutdown_connections 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.762+0000 7f57eb164740 1 -- 172.21.15.63:0/2664704360 wait complete. 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.762+0000 7f57e5a00640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 0x7f57c80a6280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x5650dc1c1dc0 tx=0x7f57d4004a60 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.762+0000 7f57eb164740 1 -- start start 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.762+0000 7f57eb164740 1 Processor -- start 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1c9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5650dc1ce630 con 0x5650dc1c0820 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5650dc0e0310 con 0x5650dc1cba00 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5650dc1ce7b0 con 0x5650dc1c1140 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57eb164740 10 client.0 _ll_register_callbacks cb 0x5650dc0aecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54830/0 (socket says 172.21.15.63:54830) 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57e5a00640 1 -- 172.21.15.63:0/2058745079 learned_addr learned my addr 172.21.15.63:0/2058745079 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57e5000640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.763+0000 7f57e6400640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 -- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 msgr2=0x5650dc1c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 -- 172.21.15.63:0/2058745079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 msgr2=0x5650dc1c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 -- 172.21.15.63:0/2058745079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650dbf05620 con 0x5650dc1c1140 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57e5000640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c8f30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0ce350 tx=0x7f57cc0d4340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4b034164599067a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57c7400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc09c6e0 con 0x5650dc1c1140 2024-04-25T18:51:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f57cc0ce5d0 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57d8004310 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57cc005180 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57eb164740 10 client.25774 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650dc1da9a0 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.764+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57d8004310 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f57cc0d8040 con 0x5650dc1c1140 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 10 client.25774 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 10 client.25774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 20 client.25774 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc1dbe60 0x5650dc1de250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 4 client.25774 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650dc13a8b0 con 0x5650dc1dbe60 2024-04-25T18:51:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.765+0000 7f57e5a00640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc1dbe60 0x5650dc1de250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.766+0000 7f57e5a00640 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc1dbe60 0x5650dc1de250 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b037353d30518b5 server_cookie=3b9c071f92c02fb2 in_seq=0 out_seq=0 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.766+0000 7f57c7400640 10 client.25774 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.766+0000 7f57c7400640 1 -- 172.21.15.63:0/2058745079 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5650dc13a8b0 con 0x5650dc1dbe60 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.766+0000 7f57c7400640 10 client.25774 handle_command_reply: tid=0 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.766+0000 7f57eb164740 1 client.25774 shutdown 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 msgr2=0x5650dc1c8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c8f30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0ce350 tx=0x7f57cc0d4340 comp rx=0 tx=0).stop 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 shutdown_connections 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc1dbe60 0x5650dc1de250 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c8f30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 --2- 172.21.15.63:0/2058745079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57c7400640 0 client.25774 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 >> 172.21.15.63:0/2058745079 conn(0x5650dc1bcd30 msgr2=0x5650dc1c2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 shutdown_connections 2024-04-25T18:51:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 1 -- 172.21.15.63:0/2058745079 wait complete. 2024-04-25T18:51:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.767+0000 7f57eb164740 20 client.25774 trim_cache size 0 max 16384 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 Processor -- start 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 -- start start 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 0x5650dc1c3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01b26d0 con 0x5650dc1cba00 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01aff70 con 0x5650dc1c0820 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.783+0000 7f57eb164740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01addc0 con 0x5650dc1c1140 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43756/0 (socket says 172.21.15.63:43756) 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49376/0 (socket says 172.21.15.63:49376) 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 -- 172.21.15.63:0/2484122379 learned_addr learned my addr 172.21.15.63:0/2484122379 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e6400640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 0x5650dc1c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 -- 172.21.15.63:0/2484122379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 msgr2=0x5650dc1d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 -- 172.21.15.63:0/2484122379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 msgr2=0x5650dc1c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 0x5650dc1c3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 -- 172.21.15.63:0/2484122379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650dc1de8b0 con 0x5650dc1c1140 2024-04-25T18:51:13.789 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5a00640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e6400640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 0x5650dc1c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57e5000640 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0d43b0 tx=0x7f57cc0048d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fb7aee1a998a2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.784+0000 7f57d2000640 1 -- 172.21.15.63:0/2484122379 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc09c500 con 0x5650dc1c1140 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57d2000640 1 -- 172.21.15.63:0/2484122379 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57cc005680 con 0x5650dc1c1140 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57d2000640 1 -- 172.21.15.63:0/2484122379 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc09cba0 con 0x5650dc1c1140 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 -- 172.21.15.63:0/2484122379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 msgr2=0x5650dc1c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57cc0d43b0 tx=0x7f57cc0048d0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 -- 172.21.15.63:0/2484122379 shutdown_connections 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1cba00 0x5650dc1c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1c9240 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 --2- 172.21.15.63:0/2484122379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1c0820 0x5650dc1d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 -- 172.21.15.63:0/2484122379 >> 172.21.15.63:0/2484122379 conn(0x5650dc1bcd30 msgr2=0x5650dc1ca220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 -- 172.21.15.63:0/2484122379 shutdown_connections 2024-04-25T18:51:13.790 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.785+0000 7f57eb164740 1 -- 172.21.15.63:0/2484122379 wait complete. 2024-04-25T18:51:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 -- start start 2024-04-25T18:51:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 Processor -- start 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01ae430 con 0x5650dc1c0820 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01af8a0 con 0x5650dc1cba00 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01aeb00 con 0x5650dc1c1140 2024-04-25T18:51:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.786+0000 7f57eb164740 10 client.0 _ll_register_callbacks cb 0x7f57cc004990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e6400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43758/0 (socket says 172.21.15.63:43758) 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 -- 172.21.15.63:0/621420517 learned_addr learned my addr 172.21.15.63:0/621420517 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5a00640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 -- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 msgr2=0x5650dc1d1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1d1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 -- 172.21.15.63:0/621420517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 msgr2=0x5650dc1d1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1d1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 -- 172.21.15.63:0/621420517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650dc241410 con 0x5650dc1c1140 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57e5000640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00c6a0 tx=0x7f57cc00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8c9ce22ced49beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57c7400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc09c500 con 0x5650dc1c1140 2024-04-25T18:51:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f57cc09c1f0 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.787+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57cc005680 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f57cc09c6e0 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 10 client.25780 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5650dc242690 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57cc005680 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f57cc09ba60 con 0x5650dc1c1140 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 10 client.25780 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 10 client.25780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 20 client.25780 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc242a80 0x5650dc244ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 4 client.25780 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5650dc134320 con 0x5650dc242a80 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.788+0000 7f57e5a00640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc242a80 0x5650dc244ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:13.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.789+0000 7f57e5a00640 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc242a80 0x5650dc244ec0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9bbe0c714583e5a server_cookie=1adc5e2e87e5e58 in_seq=0 out_seq=0 2024-04-25T18:51:13.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.789+0000 7f57c7400640 10 client.25780 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57c7400640 1 -- 172.21.15.63:0/621420517 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5650dc134320 con 0x5650dc242a80 2024-04-25T18:51:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57c7400640 10 client.25780 handle_command_reply: tid=0 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 client.25780 shutdown 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 msgr2=0x5650dc1d19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00c6a0 tx=0x7f57cc00cf90 comp rx=0 tx=0).stop 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 shutdown_connections 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5650dc242a80 0x5650dc244ec0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5650dc1cba00 0x5650dc1d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5650dc1c1140 0x5650dc1d19d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 --2- 172.21.15.63:0/621420517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5650dc1c0820 0x5650dc1d1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57c7400640 0 client.25780 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 >> 172.21.15.63:0/621420517 conn(0x5650dc1bcd30 msgr2=0x5650dc1c90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 shutdown_connections 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 1 -- 172.21.15.63:0/621420517 wait complete. 2024-04-25T18:51:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.837+0000 7f57eb164740 20 client.25780 trim_cache size 0 max 16384 2024-04-25T18:51:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:51:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:51:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:51:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 msgr2=0x7f57c80a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 0x7f57c80a6280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x5650dc1c1dc0 tx=0x7f57d4004a60 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 msgr2=0x7f57e01a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9f0 tx=0x7f57cc00edf0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 shutdown_connections 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f57c80a3dc0 0x7f57c80a6280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57e010e840 0x7f57e019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f57e0074b70 0x7f57e01a0ae0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 --2- 172.21.15.63:0/3637974575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f57e0074250 0x7f57e01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 >> 172.21.15.63:0/3637974575 conn(0x7f57e00793e0 msgr2=0x7f57e00feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 shutdown_connections 2024-04-25T18:51:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:13.838+0000 7f57e8400640 1 -- 172.21.15.63:0/3637974575 wait complete. 2024-04-25T18:51:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:14 smithi063 ceph-mon[28006]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:14.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:14 smithi070 ceph-mon[32820]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:14.454 INFO:tasks.workunit.client.0.smithi063.stdout: 1 162573 7.22 MB/sec execute 73 sec latency 40.258 ms 2024-04-25T18:51:14.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:14 smithi114 ceph-mon[32988]: pgmap v144: 97 pgs: 97 active+clean; 61 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-25T18:51:15.454 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163736 7.21 MB/sec execute 74 sec latency 26.987 ms 2024-04-25T18:51:15.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.628+0000 7f752d000700 1 -- 172.21.15.63:0/1420493446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528107960 msgr2=0x7f7528107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.628+0000 7f752d000700 1 --2- 172.21.15.63:0/1420493446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528107960 0x7f7528107d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7518009990 tx=0x7f7518009ca0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 -- 172.21.15.63:0/1420493446 shutdown_connections 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 --2- 172.21.15.63:0/1420493446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528101c50 0x7f752810f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 --2- 172.21.15.63:0/1420493446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528107960 0x7f7528107d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 --2- 172.21.15.63:0/1420493446 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101290 0x7f7528101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 -- 172.21.15.63:0/1420493446 >> 172.21.15.63:0/1420493446 conn(0x7f7528076b20 msgr2=0x7f7528076f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 -- 172.21.15.63:0/1420493446 shutdown_connections 2024-04-25T18:51:15.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.629+0000 7f752d000700 1 -- 172.21.15.63:0/1420493446 wait complete. 2024-04-25T18:51:15.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.630+0000 7f752d000700 1 Processor -- start 2024-04-25T18:51:15.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.630+0000 7f752d000700 1 -- start start 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 0x7f752819b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528107960 0x7f752819f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7528112b10 con 0x7f7528107960 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7528112990 con 0x7f7528101290 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f752d000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7528112c90 con 0x7f7528101c50 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f7527400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f7526a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 0x7f752819b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f7527400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49404/0 (socket says 172.21.15.63:49404) 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f7527400700 1 -- 172.21.15.63:0/2701838772 learned_addr learned my addr 172.21.15.63:0/2701838772 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.631+0000 7f7526a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 0x7f752819b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43772/0 (socket says 172.21.15.63:43772) 2024-04-25T18:51:15.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527e00700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528107960 0x7f752819f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 -- 172.21.15.63:0/2701838772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 msgr2=0x7f752819b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 0x7f752819b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 -- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528107960 msgr2=0x7f752819f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528107960 0x7f752819f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 -- 172.21.15.63:0/2701838772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75281a0080 con 0x7f7528101290 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.632+0000 7f7527400700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7518000c00 tx=0x7f75180057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.633+0000 7f751fe00700 1 -- 172.21.15.63:0/2701838772 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518027070 con 0x7f7528101290 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.633+0000 7f751fe00700 1 -- 172.21.15.63:0/2701838772 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7518022470 con 0x7f7528101290 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.633+0000 7f751fe00700 1 -- 172.21.15.63:0/2701838772 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518017650 con 0x7f7528101290 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.633+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75281a0310 con 0x7f7528101290 2024-04-25T18:51:15.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.633+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75281a7bf0 con 0x7f7528101290 2024-04-25T18:51:15.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.636+0000 7f751fe00700 1 -- 172.21.15.63:0/2701838772 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f751802a070 con 0x7f7528101290 2024-04-25T18:51:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.638+0000 7f751fe00700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 0x7f750c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.638+0000 7f751fe00700 1 -- 172.21.15.63:0/2701838772 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f75180d15e0 con 0x7f7528101290 2024-04-25T18:51:15.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.638+0000 7f7526a00700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 0x7f750c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 Processor -- start 2024-04-25T18:51:15.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 -- start start 2024-04-25T18:51:15.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7526a00700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 0x7f750c0a6390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7528076fd0 tx=0x7f751400b410 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b6a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1aaa020 con 0x55d9d1b60060 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1a508c0 con 0x55d9d1b6af10 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.639+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1aa5fe0 con 0x55d9d1b5f740 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43788/0 (socket says 172.21.15.63:43788) 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 -- 172.21.15.63:0/1142253609 learned_addr learned my addr 172.21.15.63:0/1142253609 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7526a00700 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b6a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 -- 172.21.15.63:0/1142253609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 msgr2=0x55d9d1b6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 -- 172.21.15.63:0/1142253609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 msgr2=0x55d9d1b6a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b6a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 -- 172.21.15.63:0/1142253609 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d1a46810 con 0x55d9d1b5f740 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f7527400700 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7518024000 tx=0x7f751809d1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b239cd8898e506f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.640+0000 7f751cc00700 1 -- 172.21.15.63:0/1142253609 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751809d5a0 con 0x55d9d1b5f740 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.641+0000 7f751cc00700 1 -- 172.21.15.63:0/1142253609 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75180a1950 con 0x55d9d1b5f740 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.641+0000 7f751cc00700 1 -- 172.21.15.63:0/1142253609 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751809d800 con 0x55d9d1b5f740 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.641+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1142253609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 msgr2=0x55d9d1b5fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.641+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7518024000 tx=0x7f751809d1b0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1142253609 shutdown_connections 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b5f740 0x55d9d1b5fb20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1142253609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b6a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1142253609 >> 172.21.15.63:0/1142253609 conn(0x55d9d1a860e0 msgr2=0x55d9d1b53360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1142253609 shutdown_connections 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1142253609 wait complete. 2024-04-25T18:51:15.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 -- start start 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.642+0000 7f7538ff7b80 1 Processor -- start 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b679f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b67f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1b6d840 con 0x55d9d1b60060 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1a639f0 con 0x55d9d1b5f740 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d9d1b6d9c0 con 0x55d9d1b6af10 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7538ff7b80 10 client.0 _ll_register_callbacks cb 0x55d9d1a428b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.643+0000 7f7527400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7526a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49422/0 (socket says 172.21.15.63:49422) 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 -- 172.21.15.63:0/4177340957 learned_addr learned my addr 172.21.15.63:0/4177340957 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527e00700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b67f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 -- 172.21.15.63:0/4177340957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 msgr2=0x55d9d1b67f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b67f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 -- 172.21.15.63:0/4177340957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 msgr2=0x55d9d1b679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 -- 172.21.15.63:0/4177340957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d19f7a60 con 0x55d9d1b5f740 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7527400700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75180a5c50 tx=0x7f75180880b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c836b0cddc6f46f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.644+0000 7f7511600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75180b6060 con 0x55d9d1b5f740 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f751809da10 con 0x55d9d1b5f740 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75180a1950 con 0x55d9d1b5f740 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f75180a6180 con 0x55d9d1b5f740 2024-04-25T18:51:15.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7538ff7b80 10 client.25775 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:15.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d9d1a40c80 con 0x55d9d1b5f740 2024-04-25T18:51:15.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75180a1950 con 0x55d9d1b5f740 2024-04-25T18:51:15.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.645+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f75180050d0 con 0x55d9d1b5f740 2024-04-25T18:51:15.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 10 client.25775 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 10 client.25775 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 20 client.25775 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1b7a340 0x55d9d1b7c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 4 client.25775 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9d1ab7810 con 0x55d9d1b7a340 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7526a00700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1b7a340 0x55d9d1b7c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7526a00700 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1b7a340 0x55d9d1b7c730 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d36d72082216e server_cookie=43cafe87f8ee7773 in_seq=0 out_seq=0 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.646+0000 7f7511600700 10 client.25775 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.647+0000 7f7511600700 1 -- 172.21.15.63:0/4177340957 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9d1ab7810 con 0x55d9d1b7a340 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.647+0000 7f7511600700 10 client.25775 handle_command_reply: tid=0 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.647+0000 7f7538ff7b80 1 client.25775 shutdown 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 msgr2=0x55d9d1b6a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f75180a5c50 tx=0x7f75180880b0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 shutdown_connections 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1b7a340 0x55d9d1b7c730 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b67f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b5f740 0x55d9d1b6a6f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7511600700 0 client.25775 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/4177340957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b60060 0x55d9d1b679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 >> 172.21.15.63:0/4177340957 conn(0x55d9d1a860e0 msgr2=0x55d9d1b52f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 shutdown_connections 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 1 -- 172.21.15.63:0/4177340957 wait complete. 2024-04-25T18:51:15.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.648+0000 7f7538ff7b80 20 client.25775 trim_cache size 0 max 16384 2024-04-25T18:51:15.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.672+0000 7f7538ff7b80 1 Processor -- start 2024-04-25T18:51:15.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.672+0000 7f7538ff7b80 1 -- start start 2024-04-25T18:51:15.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b765e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b73030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75281ad6d0 con 0x55d9d1b5f740 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75281aaf30 con 0x55d9d1b60060 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f752810d950 con 0x55d9d1b6af10 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7526a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7526a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49424/0 (socket says 172.21.15.63:49424) 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7526a00700 1 -- 172.21.15.63:0/3003070359 learned_addr learned my addr 172.21.15.63:0/3003070359 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.673+0000 7f7527e00700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b73030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 -- 172.21.15.63:0/3003070359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 msgr2=0x55d9d1b73030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b73030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7527400700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b765e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 -- 172.21.15.63:0/3003070359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 msgr2=0x55d9d1b765e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b765e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 -- 172.21.15.63:0/3003070359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9d1b68930 con 0x55d9d1b60060 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7527400700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b765e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7526a00700 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7514005d90 tx=0x7f7514013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6b5c7bed5c64969 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7513e00700 1 -- 172.21.15.63:0/3003070359 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514020bf0 con 0x55d9d1b60060 2024-04-25T18:51:15.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7513e00700 1 -- 172.21.15.63:0/3003070359 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7514020d90 con 0x55d9d1b60060 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.674+0000 7f7513e00700 1 -- 172.21.15.63:0/3003070359 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7514004bb0 con 0x55d9d1b60060 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 -- 172.21.15.63:0/3003070359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 msgr2=0x55d9d1b72af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7514005d90 tx=0x7f7514013000 comp rx=0 tx=0).stop 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 -- 172.21.15.63:0/3003070359 shutdown_connections 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b6af10 0x55d9d1b73030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b765e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/3003070359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b60060 0x55d9d1b72af0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.675+0000 7f7538ff7b80 1 -- 172.21.15.63:0/3003070359 >> 172.21.15.63:0/3003070359 conn(0x55d9d1a860e0 msgr2=0x55d9d1b525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.676+0000 7f7538ff7b80 1 -- 172.21.15.63:0/3003070359 shutdown_connections 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.676+0000 7f7538ff7b80 1 -- 172.21.15.63:0/3003070359 wait complete. 2024-04-25T18:51:15.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.676+0000 7f7538ff7b80 1 -- start start 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.676+0000 7f7538ff7b80 1 Processor -- start 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b71580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b60060 0x55d9d1b71ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b72000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75281a93f0 con 0x55d9d1b5f740 2024-04-25T18:51:15.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75281aa860 con 0x55d9d1b6af10 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f75281a9ac0 con 0x55d9d1b60060 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7538ff7b80 10 client.0 _ll_register_callbacks cb 0x55d9d1a428b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7527400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b71580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7527400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b71580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54904/0 (socket says 172.21.15.63:54904) 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7527e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b72000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7527400700 1 -- 172.21.15.63:0/1450278307 learned_addr learned my addr 172.21.15.63:0/1450278307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.677+0000 7f7526a00700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b60060 0x55d9d1b71ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 -- 172.21.15.63:0/1450278307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b60060 msgr2=0x55d9d1b71ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b60060 0x55d9d1b71ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 -- 172.21.15.63:0/1450278307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 msgr2=0x55d9d1b71580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b71580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 -- 172.21.15.63:0/1450278307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9d1be0730 con 0x55d9d1b6af10 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7527e00700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b72000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7520007f00 tx=0x7f7520007f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cffd2c78ad0cca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:15.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7511600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75200179c0 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7514020d90 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.678+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f7520017b60 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7520017d60 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 10 client.25781 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d9d1be1a30 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7514020d90 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f7520029b60 con 0x55d9d1b6af10 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 10 client.25781 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 10 client.25781 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 20 client.25781 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1be2190 0x55d9d1be45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 4 client.25781 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.35, "await_for": 42.0}] 2024-04-25T18:51:15.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.679+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.35, "await_for": 42.0}) -- 0x55d9d1ab18b0 con 0x55d9d1be2190 2024-04-25T18:51:15.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.680+0000 7f7527400700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1be2190 0x55d9d1be45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:15.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.680+0000 7f7527400700 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1be2190 0x55d9d1be45b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d69bb093b4603368 server_cookie=1f601a573f42c5b in_seq=0 out_seq=0 2024-04-25T18:51:15.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.680+0000 7f7511600700 10 client.25781 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:15.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7511600700 1 -- 172.21.15.63:0/1450278307 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55d9d1ab18b0 con 0x55d9d1be2190 2024-04-25T18:51:15.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7511600700 10 client.25781 handle_command_reply: tid=0 2024-04-25T18:51:15.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 client.25781 shutdown 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 msgr2=0x55d9d1b72000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b72000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7520007f00 tx=0x7f7520007f30 comp rx=0 tx=0).stop 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 shutdown_connections 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d9d1be2190 0x55d9d1be45b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d9d1b6af10 0x55d9d1b72000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9d1b5f740 0x55d9d1b71580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 --2- 172.21.15.63:0/1450278307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d9d1b60060 0x55d9d1b71ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7511600700 0 client.25781 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 >> 172.21.15.63:0/1450278307 conn(0x55d9d1a860e0 msgr2=0x55d9d1b52e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 shutdown_connections 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 1 -- 172.21.15.63:0/1450278307 wait complete. 2024-04-25T18:51:15.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.729+0000 7f7538ff7b80 20 client.25781 trim_cache size 0 max 16384 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 30, 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 14, 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "9213c37": { 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 14, 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:51:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 82.3, 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:15.735 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:51:15.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.730+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 msgr2=0x7f750c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.730+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 0x7f750c0a6390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7528076fd0 tx=0x7f751400b410 comp rx=0 tx=0).stop 2024-04-25T18:51:15.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.730+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 msgr2=0x7f752819b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:15.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.730+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7518000c00 tx=0x7f75180057c0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 shutdown_connections 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f750c0a3ed0 0x7f750c0a6390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7528107960 0x7f752819f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7528101290 0x7f752819b0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 --2- 172.21.15.63:0/2701838772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7528101c50 0x7f752819b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 >> 172.21.15.63:0/2701838772 conn(0x7f7528076b20 msgr2=0x7f752810dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 shutdown_connections 2024-04-25T18:51:15.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:15.731+0000 7f752d000700 1 -- 172.21.15.63:0/2701838772 wait complete. 2024-04-25T18:51:16.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:16 smithi063 ceph-mon[28006]: pgmap v145: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T18:51:16.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:16 smithi070 ceph-mon[32820]: pgmap v145: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T18:51:16.454 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 7.12 MB/sec execute 75 sec latency 735.050 ms 2024-04-25T18:51:16.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:16 smithi114 ceph-mon[32988]: pgmap v145: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 345 op/s 2024-04-25T18:51:16.720 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9213c37', quiesce took 8.6 seconds. Will release after: 22.35 2024-04-25T18:51:17.454 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 7.03 MB/sec execute 76 sec latency 1735.219 ms 2024-04-25T18:51:18.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:18 smithi070 ceph-mon[32820]: pgmap v146: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-25T18:51:18.455 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.94 MB/sec execute 77 sec latency 2735.350 ms 2024-04-25T18:51:18.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:18 smithi114 ceph-mon[32988]: pgmap v146: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-25T18:51:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:18 smithi063 ceph-mon[28006]: pgmap v146: 97 pgs: 97 active+clean; 60 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-25T18:51:19.455 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.85 MB/sec execute 78 sec latency 3736.558 ms 2024-04-25T18:51:20.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:19 smithi063 ceph-mon[28006]: pgmap v147: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 244 op/s 2024-04-25T18:51:20.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:19 smithi070 ceph-mon[32820]: pgmap v147: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 244 op/s 2024-04-25T18:51:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:19 smithi114 ceph-mon[32988]: pgmap v147: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 244 op/s 2024-04-25T18:51:20.455 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.76 MB/sec execute 79 sec latency 4736.650 ms 2024-04-25T18:51:21.455 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.68 MB/sec execute 80 sec latency 5736.812 ms 2024-04-25T18:51:22.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:22 smithi063 ceph-mon[28006]: pgmap v148: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:22.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:22 smithi070 ceph-mon[32820]: pgmap v148: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:22.455 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.59 MB/sec execute 81 sec latency 6736.971 ms 2024-04-25T18:51:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:22 smithi114 ceph-mon[32988]: pgmap v148: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:23.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.51 MB/sec execute 82 sec latency 7737.118 ms 2024-04-25T18:51:24.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:24 smithi063 ceph-mon[28006]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:24.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:24 smithi070 ceph-mon[32820]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:24.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.43 MB/sec execute 83 sec latency 8737.256 ms 2024-04-25T18:51:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:24 smithi114 ceph-mon[32988]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-25T18:51:25.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.36 MB/sec execute 84 sec latency 9737.354 ms 2024-04-25T18:51:26.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:26 smithi063 ceph-mon[28006]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-25T18:51:26.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:26 smithi070 ceph-mon[32820]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-25T18:51:26.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.28 MB/sec execute 85 sec latency 10737.511 ms 2024-04-25T18:51:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:26 smithi114 ceph-mon[32988]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-25T18:51:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 -- 172.21.15.63:0/4250907236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 msgr2=0x7f3ab01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 --2- 172.21.15.63:0/4250907236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01088d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3a98009860 tx=0x7f3a9802a340 comp rx=0 tx=0).stop 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 -- 172.21.15.63:0/4250907236 shutdown_connections 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 --2- 172.21.15.63:0/4250907236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01088d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 --2- 172.21.15.63:0/4250907236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab0115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 --2- 172.21.15.63:0/4250907236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 0x7f3ab010e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 -- 172.21.15.63:0/4250907236 >> 172.21.15.63:0/4250907236 conn(0x7f3ab0077d30 msgr2=0x7f3ab0106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 -- 172.21.15.63:0/4250907236 shutdown_connections 2024-04-25T18:51:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.994+0000 7f3aaf200640 1 -- 172.21.15.63:0/4250907236 wait complete. 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.995+0000 7f3aaf200640 1 Processor -- start 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.995+0000 7f3aaf200640 1 -- start start 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 0x7f3ab019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01185f0 con 0x7f3ab0108e10 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0118470 con 0x7f3ab0108490 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aaf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab0118770 con 0x7f3ab010e420 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aae800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 0x7f3ab019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:26.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56494/0 (socket says 172.21.15.63:56494) 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aad400640 1 -- 172.21.15.63:0/740663495 learned_addr learned my addr 172.21.15.63:0/740663495 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.996+0000 7f3aade00640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 -- 172.21.15.63:0/740663495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 msgr2=0x7f3ab019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 0x7f3ab019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 -- 172.21.15.63:0/740663495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 msgr2=0x7f3ab01a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:26.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 -- 172.21.15.63:0/740663495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab019f130 con 0x7f3ab0108e10 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aad400640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a98032d50 tx=0x7f3a98032d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aa6a00640 1 -- 172.21.15.63:0/740663495 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98037040 con 0x7f3ab0108e10 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aa6a00640 1 -- 172.21.15.63:0/740663495 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a98041070 con 0x7f3ab0108e10 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab019f3c0 con 0x7f3ab0108e10 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.997+0000 7f3aa6a00640 1 -- 172.21.15.63:0/740663495 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98003850 con 0x7f3ab0108e10 2024-04-25T18:51:27.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:26.998+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab01ab6c0 con 0x7f3ab0108e10 2024-04-25T18:51:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.001+0000 7f3ab7990740 1 Processor -- start 2024-04-25T18:51:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.001+0000 7f3aa6a00640 1 -- 172.21.15.63:0/740663495 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3a98004050 con 0x7f3ab0108e10 2024-04-25T18:51:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.001+0000 7f3ab7990740 1 -- start start 2024-04-25T18:51:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.001+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f968899b0 0x561f96889d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.001+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f96894650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f96896f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3ab7990740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f967ef900 con 0x561f96894b90 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3ab7990740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f96795ef0 con 0x561f968899b0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3ab7990740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f967eb8c0 con 0x561f9688a2d0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f96896f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f96894650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aae800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f968899b0 0x561f96889d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f96896f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56508/0 (socket says 172.21.15.63:56508) 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aad400640 1 -- 172.21.15.63:0/3841464023 learned_addr learned my addr 172.21.15.63:0/3841464023 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 -- 172.21.15.63:0/3841464023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f968899b0 msgr2=0x561f96889d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f968899b0 0x561f96889d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 -- 172.21.15.63:0/3841464023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 msgr2=0x561f96896f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f96896f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.002+0000 7f3aade00640 1 -- 172.21.15.63:0/3841464023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f96502210 con 0x561f9688a2d0 2024-04-25T18:51:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3aade00640 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f96894650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009bd0 tx=0x7f3aa0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc46d6da2c6fbf92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3aa6a00640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 0x7f3a940a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3a9f000640 1 -- 172.21.15.63:0/3841464023 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0004050 con 0x561f9688a2d0 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3a9f000640 1 -- 172.21.15.63:0/3841464023 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa00054d0 con 0x561f9688a2d0 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3a9f000640 1 -- 172.21.15.63:0/3841464023 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0004c60 con 0x561f9688a2d0 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3aa6a00640 1 -- 172.21.15.63:0/740663495 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f3a980f02b0 con 0x7f3ab0108e10 2024-04-25T18:51:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.003+0000 7f3aae800640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 0x7f3a940a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 -- 172.21.15.63:0/3841464023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 msgr2=0x561f96894650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f96894650 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009bd0 tx=0x7f3aa0011810 comp rx=0 tx=0).stop 2024-04-25T18:51:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 -- 172.21.15.63:0/3841464023 shutdown_connections 2024-04-25T18:51:27.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f96896f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f96894650 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3841464023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f968899b0 0x561f96889d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 -- 172.21.15.63:0/3841464023 >> 172.21.15.63:0/3841464023 conn(0x561f96885ec0 msgr2=0x561f968862a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3aae800640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 0x7f3a940a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005fd0 tx=0x7f3aa8010040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 -- 172.21.15.63:0/3841464023 shutdown_connections 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.004+0000 7f3ab7990740 1 -- 172.21.15.63:0/3841464023 wait complete. 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 -- start start 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 Processor -- start 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f96891b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 0x561f96892060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561f968977c0 con 0x561f96894b90 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561f967a9310 con 0x561f9688a2d0 2024-04-25T18:51:27.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.005+0000 7f3ab7990740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561f96897940 con 0x561f968899b0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3ab7990740 10 client.0 _ll_register_callbacks cb 0x561f96777cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aae800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aade00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f96891b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aae800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56520/0 (socket says 172.21.15.63:56520) 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aae800640 1 -- 172.21.15.63:0/3195208682 learned_addr learned my addr 172.21.15.63:0/3195208682 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 0x561f96892060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 -- 172.21.15.63:0/3195208682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 msgr2=0x561f96891b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f96891b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 -- 172.21.15.63:0/3195208682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 msgr2=0x561f968925a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aad400640 1 -- 172.21.15.63:0/3195208682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f965ce620 con 0x561f9688a2d0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.006+0000 7f3aae800640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3aad400640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 0x561f96892060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x561f968872e0 tx=0x7f3a980f3950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=674c8fe389c4b477 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:27.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a980bc150 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f3a980c4cd0 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa00054d0 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a980ede20 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3ab7990740 10 client.25793 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f968a3a60 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa00054d0 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.007+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f3a980b80d0 con 0x561f9688a2d0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 10 client.25793 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 10 client.25793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 20 client.25793 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f968a4f10 0x561f968a7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 4 client.25793 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f968038b0 con 0x561f968a4f10 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.008+0000 7f3aade00640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f968a4f10 0x561f968a7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.009+0000 7f3aade00640 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f968a4f10 0x561f968a7350 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eacfa07f1f2924b4 server_cookie=716c68de4301e868 in_seq=0 out_seq=0 2024-04-25T18:51:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.009+0000 7f3a93400640 10 client.25793 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.009+0000 7f3a93400640 1 -- 172.21.15.63:0/3195208682 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f968038b0 con 0x561f968a4f10 2024-04-25T18:51:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.009+0000 7f3a93400640 10 client.25793 handle_command_reply: tid=0 2024-04-25T18:51:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.009+0000 7f3ab7990740 1 client.25793 shutdown 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 msgr2=0x561f96892060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 0x561f96892060 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x561f968872e0 tx=0x7f3a980f3950 comp rx=0 tx=0).stop 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 shutdown_connections 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f968a4f10 0x561f968a7350 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f9688a2d0 0x561f96892060 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f96894b90 0x561f968925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3a93400640 0 client.25793 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3195208682 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f96891b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 >> 172.21.15.63:0/3195208682 conn(0x561f96885ec0 msgr2=0x561f9688bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 shutdown_connections 2024-04-25T18:51:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 1 -- 172.21.15.63:0/3195208682 wait complete. 2024-04-25T18:51:27.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.010+0000 7f3ab7990740 20 client.25793 trim_cache size 0 max 16384 2024-04-25T18:51:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.026+0000 7f3ab7990740 1 Processor -- start 2024-04-25T18:51:27.027 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.026+0000 7f3ab7990740 1 -- start start 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f968a0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9688ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b6800 con 0x561f9688a2d0 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b40a0 con 0x561f96894b90 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3ab7990740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b1ef0 con 0x561f968899b0 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3aad400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56528/0 (socket says 172.21.15.63:56528) 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3aad400640 1 -- 172.21.15.63:0/807800673 learned_addr learned my addr 172.21.15.63:0/807800673 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:27.029 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3aade00640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f968a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.027+0000 7f3aae800640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9688ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 -- 172.21.15.63:0/807800673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 msgr2=0x561f9688ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9688ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 -- 172.21.15.63:0/807800673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 msgr2=0x561f968857a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 -- 172.21.15.63:0/807800673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f96885ce0 con 0x561f968899b0 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aad400640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:27.030 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3aade00640 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f968a0680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0004880 tx=0x7f3aa0009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756b3ab1d009ce24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3a9e600640 1 -- 172.21.15.63:0/807800673 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0033070 con 0x561f968899b0 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3a9e600640 1 -- 172.21.15.63:0/807800673 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa002e8c0 con 0x561f968899b0 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.028+0000 7f3a9e600640 1 -- 172.21.15.63:0/807800673 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0019690 con 0x561f968899b0 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 -- 172.21.15.63:0/807800673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 msgr2=0x561f968a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f968a0680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0004880 tx=0x7f3aa0009620 comp rx=0 tx=0).stop 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 -- 172.21.15.63:0/807800673 shutdown_connections 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9688ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f9688a2d0 0x561f968857a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 --2- 172.21.15.63:0/807800673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f968899b0 0x561f968a0680 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 -- 172.21.15.63:0/807800673 >> 172.21.15.63:0/807800673 conn(0x561f96885ec0 msgr2=0x561f9688b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 -- 172.21.15.63:0/807800673 shutdown_connections 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.029+0000 7f3ab7990740 1 -- 172.21.15.63:0/807800673 wait complete. 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 -- start start 2024-04-25T18:51:27.034 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 Processor -- start 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9689b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b2560 con 0x561f968899b0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b39d0 con 0x561f96894b90 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab01b2c30 con 0x561f9688a2d0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.030+0000 7f3ab7990740 10 client.0 _ll_register_callbacks cb 0x7f3a980f3a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aade00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54694/0 (socket says 172.21.15.63:54694) 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aade00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56534/0 (socket says 172.21.15.63:56534) 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 -- 172.21.15.63:0/3226148418 learned_addr learned my addr 172.21.15.63:0/3226148418 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aae800640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9689b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.035 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 -- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 msgr2=0x561f9689b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9689b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 -- 172.21.15.63:0/3226148418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 msgr2=0x561f9689a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 -- 172.21.15.63:0/3226148418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9690a6e0 con 0x561f9688a2d0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aae800640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9689b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aade00640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3aad400640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a980bc310 tx=0x7f3a980c4d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f29557188e81a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.031+0000 7f3a93400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a980bbce0 con 0x561f9688a2d0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f3a980bb290 con 0x561f9688a2d0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3aa002e8c0 con 0x561f9688a2d0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3a980bbec0 con 0x561f9688a2d0 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3ab7990740 10 client.25816 fetch_fsmap learned FSMap version 30 2024-04-25T18:51:27.036 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561f9690a6e0 con 0x561f9688a2d0 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa002e8c0 con 0x561f9688a2d0 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.032+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 30) ==== 8342+0+0 (secure 0 0 0) 0x7f3a980c0120 con 0x561f9688a2d0 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 10 client.25816 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 10 client.25816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 20 client.25816 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f9690bd60 0x561f9690e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 4 client.25816 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f967fd320 con 0x561f9690bd60 2024-04-25T18:51:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3aade00640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f9690bd60 0x561f9690e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3aade00640 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f9690bd60 0x561f9690e180 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fba13e02bf5b4a1d server_cookie=a9e72384e4000eea in_seq=0 out_seq=0 2024-04-25T18:51:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.033+0000 7f3a93400640 10 client.25816 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.065+0000 7f3a93400640 1 -- 172.21.15.63:0/3226148418 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f967fd320 con 0x561f9690bd60 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3a93400640 10 client.25816 handle_command_reply: tid=0 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 client.25816 shutdown 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 msgr2=0x561f9689acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a980bc310 tx=0x7f3a980c4d00 comp rx=0 tx=0).stop 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 shutdown_connections 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561f9690bd60 0x561f9690e180 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561f96894b90 0x561f9689b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561f9688a2d0 0x561f9689acf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.068 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 --2- 172.21.15.63:0/3226148418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561f968899b0 0x561f9689a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3a93400640 0 client.25816 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 >> 172.21.15.63:0/3226148418 conn(0x561f96885ec0 msgr2=0x561f9688bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 shutdown_connections 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 1 -- 172.21.15.63:0/3226148418 wait complete. 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.066+0000 7f3ab7990740 20 client.25816 trim_cache size 0 max 16384 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:51:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:51:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:51:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 msgr2=0x7f3a940a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 0x7f3a940a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005fd0 tx=0x7f3aa8010040 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 msgr2=0x7f3ab01a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a98032d50 tx=0x7f3a98032d80 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 shutdown_connections 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3a940a3d90 0x7f3a940a6250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3ab010e420 0x7f3ab019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ab0108e10 0x7f3ab01a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 --2- 172.21.15.63:0/740663495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3ab0108490 0x7f3ab01a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 >> 172.21.15.63:0/740663495 conn(0x7f3ab0077d30 msgr2=0x7f3ab0078e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.067+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 shutdown_connections 2024-04-25T18:51:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:27.068+0000 7f3aaf200640 1 -- 172.21.15.63:0/740663495 wait complete. 2024-04-25T18:51:27.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.21 MB/sec execute 86 sec latency 11737.669 ms 2024-04-25T18:51:28.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:28 smithi063 ceph-mon[28006]: pgmap v151: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:51:28.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:28 smithi070 ceph-mon[32820]: pgmap v151: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:51:28.456 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.14 MB/sec execute 87 sec latency 12737.814 ms 2024-04-25T18:51:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:28 smithi114 ceph-mon[32988]: pgmap v151: 97 pgs: 97 active+clean; 62 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:51:29.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.07 MB/sec execute 88 sec latency 13737.953 ms 2024-04-25T18:51:30.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:30 smithi063 ceph-mon[28006]: pgmap v152: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-25T18:51:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:30 smithi070 ceph-mon[32820]: pgmap v152: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-25T18:51:30.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 6.00 MB/sec execute 89 sec latency 14738.131 ms 2024-04-25T18:51:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:30 smithi114 ceph-mon[32988]: pgmap v152: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-25T18:51:31.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.93 MB/sec execute 90 sec latency 15738.272 ms 2024-04-25T18:51:32.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:32 smithi063 ceph-mon[28006]: pgmap v153: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:32.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:32 smithi070 ceph-mon[32820]: pgmap v153: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:32.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.87 MB/sec execute 91 sec latency 16738.430 ms 2024-04-25T18:51:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:32 smithi114 ceph-mon[32988]: pgmap v153: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:33.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.80 MB/sec execute 92 sec latency 17738.574 ms 2024-04-25T18:51:34.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:34 smithi063 ceph-mon[28006]: pgmap v154: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:34.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:34 smithi070 ceph-mon[32820]: pgmap v154: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:34.457 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.74 MB/sec execute 93 sec latency 18738.730 ms 2024-04-25T18:51:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:34 smithi114 ceph-mon[32988]: pgmap v154: 97 pgs: 97 active+clean; 61 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 998 KiB/s wr, 56 op/s 2024-04-25T18:51:35.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.68 MB/sec execute 94 sec latency 19738.871 ms 2024-04-25T18:51:36.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:36 smithi063 ceph-mon[28006]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-04-25T18:51:36.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:36 smithi070 ceph-mon[32820]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-04-25T18:51:36.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.62 MB/sec execute 95 sec latency 20739.039 ms 2024-04-25T18:51:36.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:36 smithi114 ceph-mon[32988]: pgmap v155: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 1.2 MiB/s wr, 74 op/s 2024-04-25T18:51:37.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.56 MB/sec execute 96 sec latency 21739.225 ms 2024-04-25T18:51:38.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:38 smithi063 ceph-mon[28006]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 730 KiB/s wr, 50 op/s 2024-04-25T18:51:38.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:38 smithi070 ceph-mon[32820]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 730 KiB/s wr, 50 op/s 2024-04-25T18:51:38.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.51 MB/sec execute 97 sec latency 22739.390 ms 2024-04-25T18:51:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:38 smithi114 ceph-mon[32988]: pgmap v156: 97 pgs: 97 active+clean; 57 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 730 KiB/s wr, 50 op/s 2024-04-25T18:51:39.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:51:39.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.45 MB/sec execute 98 sec latency 23739.455 ms 2024-04-25T18:51:39.537 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 -- 172.21.15.63:0/1694908989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 msgr2=0x7f7364104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.219 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 --2- 172.21.15.63:0/1694908989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f7364104790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7354009900 tx=0x7f735402a3f0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 -- 172.21.15.63:0/1694908989 shutdown_connections 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 --2- 172.21.15.63:0/1694908989 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7364109480 0x7f7364111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 --2- 172.21.15.63:0/1694908989 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364104d60 0x7f7364108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 --2- 172.21.15.63:0/1694908989 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f7364104790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 -- 172.21.15.63:0/1694908989 >> 172.21.15.63:0/1694908989 conn(0x7f73640ffc50 msgr2=0x7f7364102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 -- 172.21.15.63:0/1694908989 shutdown_connections 2024-04-25T18:51:40.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.218+0000 7f7369000640 1 -- 172.21.15.63:0/1694908989 wait complete. 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.219+0000 7f7369000640 1 Processor -- start 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.219+0000 7f7369000640 1 -- start start 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f73641a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 0x7f73641a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 0x7f736419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7364114410 con 0x7f7364109480 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7364114290 con 0x7f73641043b0 2024-04-25T18:51:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7369000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7364114590 con 0x7f736419a6f0 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 0x7f73641a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7361e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f73641a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7362800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 0x7f736419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 0x7f73641a0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48120/0 (socket says 172.21.15.63:48120) 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7361e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f73641a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56232/0 (socket says 172.21.15.63:56232) 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.220+0000 7f7361400640 1 -- 172.21.15.63:0/886791532 learned_addr learned my addr 172.21.15.63:0/886791532 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 -- 172.21.15.63:0/886791532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 msgr2=0x7f73641a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f73641a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 -- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 msgr2=0x7f73641a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 0x7f73641a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 -- 172.21.15.63:0/886791532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736419b2f0 con 0x7f736419a6f0 2024-04-25T18:51:40.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7362800640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 0x7f736419ab70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f735c009fd0 tx=0x7f735c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f735aa00640 1 -- 172.21.15.63:0/886791532 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c00ce20 con 0x7f736419a6f0 2024-04-25T18:51:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f735aa00640 1 -- 172.21.15.63:0/886791532 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f735c005020 con 0x7f736419a6f0 2024-04-25T18:51:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f735aa00640 1 -- 172.21.15.63:0/886791532 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c010690 con 0x7f736419a6f0 2024-04-25T18:51:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73641a73d0 con 0x7f736419a6f0 2024-04-25T18:51:40.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.221+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73641a77d0 con 0x7f736419a6f0 2024-04-25T18:51:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.225+0000 7f736bf9f740 1 Processor -- start 2024-04-25T18:51:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.226+0000 7f735aa00640 1 -- 172.21.15.63:0/886791532 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f735c0054d0 con 0x7f736419a6f0 2024-04-25T18:51:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 -- start start 2024-04-25T18:51:40.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99def8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df05df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d99de5e900 con 0x55d99def9140 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d99de04ef0 con 0x55d99def8820 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f736bf9f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d99de5a8c0 con 0x55d99df03a00 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f7361e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f7361e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48130/0 (socket says 172.21.15.63:48130) 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f7361e00640 1 -- 172.21.15.63:0/1685596330 learned_addr learned my addr 172.21.15.63:0/1685596330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.227+0000 7f7361400640 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99def8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361400640 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df05df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 -- 172.21.15.63:0/1685596330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 msgr2=0x55d99df05df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df05df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 -- 172.21.15.63:0/1685596330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 msgr2=0x55d99def8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99def8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 -- 172.21.15.63:0/1685596330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d99db71210 con 0x55d99def9140 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f7361e00640 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7354000c00 tx=0x7f7354004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c6dfee38152921 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.228+0000 7f735aa00640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 0x7f73480a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.229+0000 7f735aa00640 1 -- 172.21.15.63:0/886791532 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f735c0d0dd0 con 0x7f736419a6f0 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.229+0000 7f734f400640 1 -- 172.21.15.63:0/1685596330 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354035040 con 0x55d99def9140 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.229+0000 7f734f400640 1 -- 172.21.15.63:0/1685596330 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73540054f0 con 0x55d99def9140 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.229+0000 7f7361400640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 0x7f73480a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.229+0000 7f734f400640 1 -- 172.21.15.63:0/1685596330 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73540035f0 con 0x55d99def9140 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 -- 172.21.15.63:0/1685596330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 msgr2=0x55d99df034c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7354000c00 tx=0x7f7354004040 comp rx=0 tx=0).stop 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f7361400640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 0x7f73480a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55d99def6150 tx=0x7f7350009530 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 -- 172.21.15.63:0/1685596330 shutdown_connections 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df05df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df034c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1685596330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99def8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 -- 172.21.15.63:0/1685596330 >> 172.21.15.63:0/1685596330 conn(0x55d99def4d30 msgr2=0x55d99def5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 -- 172.21.15.63:0/1685596330 shutdown_connections 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.230+0000 7f736bf9f740 1 -- 172.21.15.63:0/1685596330 wait complete. 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 -- start start 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 Processor -- start 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df00980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df00ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df01400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d99df065d0 con 0x55d99def9140 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d99de18310 con 0x55d99def8820 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.231+0000 7f736bf9f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d99df06750 con 0x55d99df03a00 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f736bf9f740 10 client.0 _ll_register_callbacks cb 0x55d99dde6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df00ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df00ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48132/0 (socket says 172.21.15.63:48132) 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7361400640 1 -- 172.21.15.63:0/568675641 learned_addr learned my addr 172.21.15.63:0/568675641 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df01400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7361e00640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df00980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 -- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 msgr2=0x55d99df00980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df00980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 -- 172.21.15.63:0/568675641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 msgr2=0x55d99df00ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df00ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 -- 172.21.15.63:0/568675641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d99dc3d620 con 0x55d99df03a00 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.232+0000 7f7362800640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df01400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f735c0ce510 tx=0x7f735c09d5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306fc80ab3ade784 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0b5020 con 0x55d99df03a00 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f735c0993d0 con 0x55d99df03a00 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73540054f0 con 0x55d99df03a00 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f735c0a59d0 con 0x55d99df03a00 2024-04-25T18:51:40.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f736bf9f740 10 client.25828 fetch_fsmap learned FSMap version 31 2024-04-25T18:51:40.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d99df128b0 con 0x55d99df03a00 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73540054f0 con 0x55d99df03a00 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.233+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f735c0bd0f0 con 0x55d99df03a00 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 10 client.25828 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 10 client.25828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 20 client.25828 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df13d70 0x55d99df161a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 4 client.25828 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d99de728b0 con 0x55d99df13d70 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.234+0000 7f7361e00640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df13d70 0x55d99df161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.235+0000 7f7343e00640 10 client.25828 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.235+0000 7f7361e00640 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df13d70 0x55d99df161a0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df5d109f306be236 server_cookie=d81d53a57a4b4c82 in_seq=0 out_seq=0 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.235+0000 7f7343e00640 1 -- 172.21.15.63:0/568675641 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d99de728b0 con 0x55d99df13d70 2024-04-25T18:51:40.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.235+0000 7f7343e00640 10 client.25828 handle_command_reply: tid=0 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 client.25828 shutdown 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 msgr2=0x55d99df01400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df01400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f735c0ce510 tx=0x7f735c09d5a0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 shutdown_connections 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df13d70 0x55d99df161a0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df01400 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df00ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 --2- 172.21.15.63:0/568675641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df00980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f7343e00640 0 client.25828 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 >> 172.21.15.63:0/568675641 conn(0x55d99def4d30 msgr2=0x55d99defaab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 shutdown_connections 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 1 -- 172.21.15.63:0/568675641 wait complete. 2024-04-25T18:51:40.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.236+0000 7f736bf9f740 20 client.25828 trim_cache size 0 max 16384 2024-04-25T18:51:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.268+0000 7f736bf9f740 1 Processor -- start 2024-04-25T18:51:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.268+0000 7f736bf9f740 1 -- start start 2024-04-25T18:51:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def9140 0x55d99df1bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df1bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73641b26b0 con 0x55d99def8820 2024-04-25T18:51:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73641afef0 con 0x55d99def9140 2024-04-25T18:51:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f736bf9f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73641add40 con 0x55d99df03a00 2024-04-25T18:51:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f7361e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f7361e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48136/0 (socket says 172.21.15.63:48136) 2024-04-25T18:51:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f7361e00640 1 -- 172.21.15.63:0/1376242389 learned_addr learned my addr 172.21.15.63:0/1376242389 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:40.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f7361400640 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def9140 0x55d99df1bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.269+0000 7f7362800640 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df1bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 -- 172.21.15.63:0/1376242389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 msgr2=0x55d99df1bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df1bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 -- 172.21.15.63:0/1376242389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def9140 msgr2=0x55d99df1bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def9140 0x55d99df1bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 -- 172.21.15.63:0/1376242389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d99def0690 con 0x55d99def8820 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f7361e00640 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7354009230 tx=0x7f7354004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a4674bce6215c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f734ea00640 1 -- 172.21.15.63:0/1376242389 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354035040 con 0x55d99def8820 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f734ea00640 1 -- 172.21.15.63:0/1376242389 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73540545d0 con 0x55d99def8820 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.270+0000 7f734ea00640 1 -- 172.21.15.63:0/1376242389 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73540039c0 con 0x55d99def8820 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 -- 172.21.15.63:0/1376242389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 msgr2=0x55d99df0f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7354009230 tx=0x7f7354004ab0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 -- 172.21.15.63:0/1376242389 shutdown_connections 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def9140 0x55d99df1bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df1bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 --2- 172.21.15.63:0/1376242389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def8820 0x55d99df0f490 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.271+0000 7f736bf9f740 1 -- 172.21.15.63:0/1376242389 >> 172.21.15.63:0/1376242389 conn(0x55d99def4d30 msgr2=0x55d99def5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.272+0000 7f736bf9f740 1 -- 172.21.15.63:0/1376242389 shutdown_connections 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.272+0000 7f736bf9f740 1 -- 172.21.15.63:0/1376242389 wait complete. 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.272+0000 7f736bf9f740 1 -- start start 2024-04-25T18:51:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.272+0000 7f736bf9f740 1 Processor -- start 2024-04-25T18:51:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df0bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73641ae3b0 con 0x55d99def9140 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f73641af820 con 0x55d99def8820 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f73641aea80 con 0x55d99df03a00 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f736bf9f740 10 client.0 _ll_register_callbacks cb 0x7f735c09c6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.273+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7361400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48152/0 (socket says 172.21.15.63:48152) 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7361400640 1 -- 172.21.15.63:0/2106210399 learned_addr learned my addr 172.21.15.63:0/2106210399 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7361e00640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df0bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 -- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 msgr2=0x55d99df0bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df0bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 -- 172.21.15.63:0/2106210399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 msgr2=0x55d99df08f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 -- 172.21.15.63:0/2106210399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d99df09b20 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7361400640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7362800640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df094b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f735c0bd550 tx=0x7f735c09cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3cd17fc2c924c96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.274+0000 7f7343e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0b5040 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f735c098bd0 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73540545d0 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f735c099000 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f736bf9f740 10 client.25834 fetch_fsmap learned FSMap version 31 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d99df82950 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73540545d0 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.275+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f735c0a55b0 con 0x55d99df03a00 2024-04-25T18:51:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 10 client.25834 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 10 client.25834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 20 client.25834 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df83130 0x55d99df85550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 4 client.25834 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d99de6c320 con 0x55d99df83130 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f7361e00640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df83130 0x55d99df85550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f7343e00640 10 client.25834 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:40.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.276+0000 7f7361e00640 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df83130 0x55d99df85550 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a67fdc9f0edddaf0 server_cookie=4740160863ce07a4 in_seq=0 out_seq=0 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f7343e00640 1 -- 172.21.15.63:0/2106210399 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d99de6c320 con 0x55d99df83130 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f7343e00640 10 client.25834 handle_command_reply: tid=0 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 client.25834 shutdown 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 msgr2=0x55d99df094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df094b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f735c0bd550 tx=0x7f735c09cea0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 shutdown_connections 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d99df83130 0x55d99df85550 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d99df03a00 0x55d99df094b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d99def9140 0x55d99df08f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f7343e00640 0 client.25834 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 --2- 172.21.15.63:0/2106210399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d99def8820 0x55d99df0bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 >> 172.21.15.63:0/2106210399 conn(0x55d99def4d30 msgr2=0x55d99def3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 shutdown_connections 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.303+0000 7f736bf9f740 1 -- 172.21.15.63:0/2106210399 wait complete. 2024-04-25T18:51:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f736bf9f740 20 client.25834 trim_cache size 0 max 16384 2024-04-25T18:51:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:51:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:51:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:51:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 msgr2=0x7f73480a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 0x7f73480a6300 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x55d99def6150 tx=0x7f7350009530 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 msgr2=0x7f736419ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 0x7f736419ab70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f735c009fd0 tx=0x7f735c00ef10 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 shutdown_connections 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f73480a3e40 0x7f73480a6300 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f736419a6f0 0x7f736419ab70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7364109480 0x7f73641a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 --2- 172.21.15.63:0/886791532 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f73641043b0 0x7f73641a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.304+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 >> 172.21.15.63:0/886791532 conn(0x7f73640ffc50 msgr2=0x7f7364101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.305+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 shutdown_connections 2024-04-25T18:51:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:40.305+0000 7f7369000640 1 -- 172.21.15.63:0/886791532 wait complete. 2024-04-25T18:51:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:40 smithi070 ceph-mon[32820]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-04-25T18:51:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:40 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:40.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.39 MB/sec execute 99 sec latency 24739.592 ms 2024-04-25T18:51:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:40 smithi114 ceph-mon[32988]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-04-25T18:51:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:40 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:40 smithi063 ceph-mon[28006]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.0 MiB/s wr, 69 op/s 2024-04-25T18:51:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:40 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:41.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.34 MB/sec execute 100 sec latency 25739.686 ms 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 -- 172.21.15.63:0/1943227216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d0109c80 msgr2=0x7fc8d010dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 --2- 172.21.15.63:0/1943227216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d0109c80 0x7fc8d010dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc8bc009990 tx=0x7fc8bc009ca0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 -- 172.21.15.63:0/1943227216 shutdown_connections 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 --2- 172.21.15.63:0/1943227216 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 0x7fc8d0115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 --2- 172.21.15.63:0/1943227216 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d01092d0 0x7fc8d01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 --2- 172.21.15.63:0/1943227216 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d0109c80 0x7fc8d010dcd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 -- 172.21.15.63:0/1943227216 >> 172.21.15.63:0/1943227216 conn(0x7fc8d0076210 msgr2=0x7fc8d0078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 -- 172.21.15.63:0/1943227216 shutdown_connections 2024-04-25T18:51:42.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.141+0000 7fc8d8200700 1 -- 172.21.15.63:0/1943227216 wait complete. 2024-04-25T18:51:42.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.142+0000 7fc8d8200700 1 Processor -- start 2024-04-25T18:51:42.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.142+0000 7fc8d8200700 1 -- start start 2024-04-25T18:51:42.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:42.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0109c80 0x7fc8d01a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:42.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 0x7fc8d019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0119290 con 0x7fc8d0109c80 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0119110 con 0x7fc8d01092d0 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d8200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d0119410 con 0x7fc8d010e3b0 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d6200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 0x7fc8d019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d4e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0109c80 0x7fc8d01a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56276/0 (socket says 172.21.15.63:56276) 2024-04-25T18:51:42.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 -- 172.21.15.63:0/2335998803 learned_addr learned my addr 172.21.15.63:0/2335998803 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:42.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 -- 172.21.15.63:0/2335998803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 msgr2=0x7fc8d019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 0x7fc8d019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 -- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0109c80 msgr2=0x7fc8d01a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0109c80 0x7fc8d01a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.143+0000 7fc8d5800700 1 -- 172.21.15.63:0/2335998803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d019fd70 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8d5800700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400eb30 tx=0x7fc8c400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c400cbd0 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8c40044d0 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c4003e30 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d01a0060 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.144+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d01ac320 con 0x7fc8d01092d0 2024-04-25T18:51:42.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.145+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d0069730 con 0x7fc8d01092d0 2024-04-25T18:51:42.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.147+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc8c4004000 con 0x7fc8d01092d0 2024-04-25T18:51:42.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.149+0000 7fc8ce000700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 0x7fc8b80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:42.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.149+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fc8c40d0b70 con 0x7fc8d01092d0 2024-04-25T18:51:42.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.149+0000 7fc8d4e00700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 0x7fc8b80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:42.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.150+0000 7fc8d4e00700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 0x7fc8b80a62f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0108330 tx=0x7fc8bc005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:42.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.153+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8c40042f0 con 0x7fc8d01092d0 2024-04-25T18:51:42.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:42 smithi063 ceph-mon[28006]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:42.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.436+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d002df80 con 0x7fc8d01092d0 2024-04-25T18:51:42.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.437+0000 7fc8ce000700 1 -- 172.21.15.63:0/2335998803 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11872 (secure 0 0 0) 0x7fc8c409d1a0 con 0x7fc8d01092d0 2024-04-25T18:51:42.439 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:51:42.439 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:51:38.994954+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.440+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 msgr2=0x7fc8b80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.440+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 0x7fc8b80a62f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0108330 tx=0x7fc8bc005fd0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.440+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 msgr2=0x7fc8d01a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.440+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc8c400eb30 tx=0x7fc8c400ee40 comp rx=0 tx=0).stop 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.440+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 shutdown_connections 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8b80a3e30 0x7fc8b80a62f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc8d010e3b0 0x7fc8d019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8d01092d0 0x7fc8d01a5450 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 --2- 172.21.15.63:0/2335998803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8d0109c80 0x7fc8d01a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:42.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 >> 172.21.15.63:0/2335998803 conn(0x7fc8d0076210 msgr2=0x7fc8d0078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:42.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 shutdown_connections 2024-04-25T18:51:42.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:42.441+0000 7fc8d8200700 1 -- 172.21.15.63:0/2335998803 wait complete. 2024-04-25T18:51:42.445 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 31 2024-04-25T18:51:42.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:42 smithi070 ceph-mon[32820]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:42.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.29 MB/sec execute 101 sec latency 26739.761 ms 2024-04-25T18:51:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:42 smithi114 ceph-mon[32988]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:43.401 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:43 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2335998803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:43.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:43 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2335998803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:43.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.24 MB/sec execute 102 sec latency 27739.874 ms 2024-04-25T18:51:43.467 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id 9213c37 --release --await' 2024-04-25T18:51:43.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id 9213c37 --release --await 2024-04-25T18:51:43.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:43 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2335998803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:43.938 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:44.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:44 smithi063 ceph-mon[28006]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:44.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:44 smithi070 ceph-mon[32820]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:44.458 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.18 MB/sec execute 103 sec latency 28739.990 ms 2024-04-25T18:51:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:44 smithi114 ceph-mon[32988]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 580 KiB/s wr, 36 op/s 2024-04-25T18:51:45.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.13 MB/sec execute 104 sec latency 29740.120 ms 2024-04-25T18:51:46.239 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:46 smithi063 ceph-mon[28006]: pgmap v160: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 786 KiB/s wr, 57 op/s 2024-04-25T18:51:46.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:46 smithi070 ceph-mon[32820]: pgmap v160: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 786 KiB/s wr, 57 op/s 2024-04-25T18:51:46.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 163975 5.09 MB/sec execute 105 sec latency 30740.237 ms 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 -- 172.21.15.63:0/2698496576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714074c70 msgr2=0x7f5714075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 --2- 172.21.15.63:0/2698496576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714074c70 0x7f5714075050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f570400b0a0 tx=0x7f570400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 -- 172.21.15.63:0/2698496576 shutdown_connections 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 --2- 172.21.15.63:0/2698496576 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f5714116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 --2- 172.21.15.63:0/2698496576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714074c70 0x7f5714075050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 --2- 172.21.15.63:0/2698496576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714075590 0x7f57141136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 -- 172.21.15.63:0/2698496576 >> 172.21.15.63:0/2698496576 conn(0x7f5714079ba0 msgr2=0x7f5714079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 -- 172.21.15.63:0/2698496576 shutdown_connections 2024-04-25T18:51:46.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.532+0000 7f5719000700 1 -- 172.21.15.63:0/2698496576 wait complete. 2024-04-25T18:51:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.534+0000 7f5719000700 1 Processor -- start 2024-04-25T18:51:46.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.534+0000 7f5719000700 1 -- start start 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714075590 0x7f57141a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f571419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5714119260 con 0x7f5714074c70 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57141190e0 con 0x7f5714075590 2024-04-25T18:51:46.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5719000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57141193e0 con 0x7f5714113c10 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5711e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5711e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48176/0 (socket says 172.21.15.63:48176) 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5711e00700 1 -- 172.21.15.63:0/3042112751 learned_addr learned my addr 172.21.15.63:0/3042112751 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5712800700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f571419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.535+0000 7f5709400700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714075590 0x7f57141a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 -- 172.21.15.63:0/3042112751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 msgr2=0x7f571419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f571419f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 -- 172.21.15.63:0/3042112751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714075590 msgr2=0x7f57141a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714075590 0x7f57141a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 -- 172.21.15.63:0/3042112751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571419fd20 con 0x7f5714074c70 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5712800700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f571419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5711e00700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f570400d010 tx=0x7f5704004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f570b400700 1 -- 172.21.15.63:0/3042112751 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704021070 con 0x7f5714074c70 2024-04-25T18:51:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f570b400700 1 -- 172.21.15.63:0/3042112751 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5704007e30 con 0x7f5714074c70 2024-04-25T18:51:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f570b400700 1 -- 172.21.15.63:0/3042112751 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704004640 con 0x7f5714074c70 2024-04-25T18:51:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.536+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571419ffb0 con 0x7f5714074c70 2024-04-25T18:51:46.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.537+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57141ac2d0 con 0x7f5714074c70 2024-04-25T18:51:46.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.541+0000 7f570b400700 1 -- 172.21.15.63:0/3042112751 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5704007980 con 0x7f5714074c70 2024-04-25T18:51:46.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 Processor -- start 2024-04-25T18:51:46.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 -- start start 2024-04-25T18:51:46.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b979660 0x563a1b983f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9868c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f570b400700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 0x7f56f80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563a1b8cbf90 con 0x563a1b9844d0 2024-04-25T18:51:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a1b872830 con 0x563a1b979660 2024-04-25T18:51:46.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f57239a3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a1b8c7f50 con 0x563a1b978d40 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.542+0000 7f570b400700 1 -- 172.21.15.63:0/3042112751 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f570400f020 con 0x7f5714074c70 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42478/0 (socket says 172.21.15.63:42478) 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 -- 172.21.15.63:0/134539180 learned_addr learned my addr 172.21.15.63:0/134539180 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5712800700 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9868c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5709400700 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b979660 0x563a1b983f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 -- 172.21.15.63:0/134539180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b979660 msgr2=0x563a1b983f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b979660 0x563a1b983f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5712800700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 0x7f56f80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 -- 172.21.15.63:0/134539180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 msgr2=0x563a1b9868c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9868c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 -- 172.21.15.63:0/134539180 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1b868780 con 0x563a1b978d40 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.543+0000 7f5711e00700 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f57040a6350 tx=0x7f570400b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c5f8914d6c4204c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.544+0000 7f5703400700 1 -- 172.21.15.63:0/134539180 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570409e3a0 con 0x563a1b978d40 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.544+0000 7f5703400700 1 -- 172.21.15.63:0/134539180 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570409dca0 con 0x563a1b978d40 2024-04-25T18:51:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.544+0000 7f5703400700 1 -- 172.21.15.63:0/134539180 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570409bfa0 con 0x563a1b978d40 2024-04-25T18:51:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- 172.21.15.63:0/134539180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 msgr2=0x563a1b979120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f5712800700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 0x7f56f80a6270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f570c005950 tx=0x7f570c009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f57040a6350 tx=0x7f570400b740 comp rx=0 tx=0).stop 2024-04-25T18:51:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- 172.21.15.63:0/134539180 shutdown_connections 2024-04-25T18:51:46.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b979120 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9868c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 --2- 172.21.15.63:0/134539180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b979660 0x563a1b983f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- 172.21.15.63:0/134539180 >> 172.21.15.63:0/134539180 conn(0x563a1b8a8050 msgr2=0x563a1b975250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- 172.21.15.63:0/134539180 shutdown_connections 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- 172.21.15.63:0/134539180 wait complete. 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.545+0000 7f57239a3b80 1 -- start start 2024-04-25T18:51:46.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 Processor -- start 2024-04-25T18:51:46.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b983c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 0x563a1b981480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563a1b986e00 con 0x563a1b979660 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a1b885960 con 0x563a1b9844d0 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a1b986f80 con 0x563a1b978d40 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.546+0000 7f57239a3b80 10 client.0 _ll_register_callbacks cb 0x563a1b864820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48200/0 (socket says 172.21.15.63:48200) 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 -- 172.21.15.63:0/4070989800 learned_addr learned my addr 172.21.15.63:0/4070989800 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5711e00700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b983c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5712800700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 0x563a1b981480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 -- 172.21.15.63:0/4070989800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 msgr2=0x563a1b983c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b983c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 -- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 msgr2=0x563a1b981480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 0x563a1b981480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5709400700 1 -- 172.21.15.63:0/4070989800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1b8199d0 con 0x563a1b979660 2024-04-25T18:51:46.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.547+0000 7f5712800700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 0x563a1b981480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f5709400700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56fc005c50 tx=0x7f56fc0129b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160b38951d32f255 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f56f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc004030 con 0x563a1b979660 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f56fc0041d0 con 0x563a1b979660 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f570409dca0 con 0x563a1b979660 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f56fc0043d0 con 0x563a1b979660 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f57239a3b80 10 client.15651 fetch_fsmap learned FSMap version 31 2024-04-25T18:51:46.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.548+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563a1b862bf0 con 0x563a1b979660 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f570409dca0 con 0x563a1b979660 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f56fc01c5a0 con 0x563a1b979660 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 10 client.15651 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 10 client.15651 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 20 client.15651 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b993a70 0x563a1b995e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 4 client.15651 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a1b8d9780 con 0x563a1b993a70 2024-04-25T18:51:46.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.549+0000 7f5711e00700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b993a70 0x563a1b995e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.550+0000 7f5711e00700 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b993a70 0x563a1b995e60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b046018b470996d4 server_cookie=2644eb38544a50bd in_seq=0 out_seq=0 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.550+0000 7f56f7e00700 10 client.15651 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f56f7e00700 1 -- 172.21.15.63:0/4070989800 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a1b8d9780 con 0x563a1b993a70 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f56f7e00700 10 client.15651 handle_command_reply: tid=0 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 client.15651 shutdown 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 msgr2=0x563a1b980f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f56fc005c50 tx=0x7f56fc0129b0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 shutdown_connections 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b993a70 0x563a1b995e60 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b978d40 0x563a1b983c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b9844d0 0x563a1b981480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 --2- 172.21.15.63:0/4070989800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b980f40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f56f7e00700 0 client.15651 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:46.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.551+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 >> 172.21.15.63:0/4070989800 conn(0x563a1b8a8050 msgr2=0x563a1b974e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.552+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 shutdown_connections 2024-04-25T18:51:46.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.552+0000 7f57239a3b80 1 -- 172.21.15.63:0/4070989800 wait complete. 2024-04-25T18:51:46.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.552+0000 7f57239a3b80 20 client.15651 trim_cache size 0 max 16384 2024-04-25T18:51:46.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:46 smithi114 ceph-mon[32988]: pgmap v160: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 786 KiB/s wr, 57 op/s 2024-04-25T18:51:46.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.573+0000 7f57239a3b80 1 Processor -- start 2024-04-25T18:51:46.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 -- start start 2024-04-25T18:51:46.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b97b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 0x563a1b99b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b6290 con 0x563a1b979660 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b3a90 con 0x563a1b978d40 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f57239a3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b18e0 con 0x563a1b9844d0 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5709400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b97b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5712800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 0x563a1b99b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5711e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5711e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56324/0 (socket says 172.21.15.63:56324) 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5712800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 0x563a1b99b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:42502/0 (socket says 172.21.15.63:42502) 2024-04-25T18:51:46.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.574+0000 7f5711e00700 1 -- 172.21.15.63:0/413297461 learned_addr learned my addr 172.21.15.63:0/413297461 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 -- 172.21.15.63:0/413297461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 msgr2=0x563a1b99b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 0x563a1b99b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 -- 172.21.15.63:0/413297461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 msgr2=0x563a1b97b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b97b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 -- 172.21.15.63:0/413297461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1b991460 con 0x563a1b978d40 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5711e00700 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f57040062a0 tx=0x7f57040be040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75fb2c0322220321 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5702a00700 1 -- 172.21.15.63:0/413297461 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57040a6880 con 0x563a1b978d40 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5702a00700 1 -- 172.21.15.63:0/413297461 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5704003db0 con 0x563a1b978d40 2024-04-25T18:51:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.575+0000 7f5702a00700 1 -- 172.21.15.63:0/413297461 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570409bfa0 con 0x563a1b978d40 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 -- 172.21.15.63:0/413297461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 msgr2=0x563a1b98fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f57040062a0 tx=0x7f57040be040 comp rx=0 tx=0).stop 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 -- 172.21.15.63:0/413297461 shutdown_connections 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b9844d0 0x563a1b99b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98fc00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 --2- 172.21.15.63:0/413297461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b979660 0x563a1b97b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 -- 172.21.15.63:0/413297461 >> 172.21.15.63:0/413297461 conn(0x563a1b8a8050 msgr2=0x563a1b97c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 -- 172.21.15.63:0/413297461 shutdown_connections 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.576+0000 7f57239a3b80 1 -- 172.21.15.63:0/413297461 wait complete. 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.577+0000 7f57239a3b80 1 -- start start 2024-04-25T18:51:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.577+0000 7f57239a3b80 1 Processor -- start 2024-04-25T18:51:46.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 0x563a1b9870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9877c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b1f50 con 0x563a1b9844d0 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b33c0 con 0x563a1b978d40 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f57141b2620 con 0x563a1b979660 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f57239a3b80 10 client.0 _ll_register_callbacks cb 0x563a1b864820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5712800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9877c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56336/0 (socket says 172.21.15.63:56336) 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5709400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 0x563a1b9870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 -- 172.21.15.63:0/3490178385 learned_addr learned my addr 172.21.15.63:0/3490178385 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 -- 172.21.15.63:0/3490178385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 msgr2=0x563a1b9870d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 0x563a1b9870d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 -- 172.21.15.63:0/3490178385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 msgr2=0x563a1b9877c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9877c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.578+0000 7f5711e00700 1 -- 172.21.15.63:0/3490178385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1b9f9f20 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f5712800700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f5709400700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 0x563a1b9870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f5711e00700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f570409d770 tx=0x7f570400b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bad5659134bd4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f56f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57040a6880 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f570409bd40 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5704003db0 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f57040047e0 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f57239a3b80 10 client.25841 fetch_fsmap learned FSMap version 31 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.579+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563a1b9fb110 con 0x563a1b978d40 2024-04-25T18:51:46.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5704003db0 con 0x563a1b978d40 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 31) ==== 8350+0+0 (secure 0 0 0) 0x7f5704011070 con 0x563a1b978d40 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 10 client.25841 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 10 client.25841 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 20 client.25841 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b9fb3e0 0x563a1b9fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 4 client.25841 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9213c37", "release": true, "await": true}] 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9213c37", "release": true, "await": true}) -- 0x563a1b8d3820 con 0x563a1b9fb3e0 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.580+0000 7f5709400700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b9fb3e0 0x563a1b9fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.581+0000 7f5709400700 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b9fb3e0 0x563a1b9fd7d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=830f8c9e5775d1a6 server_cookie=a11f49ec95e21653 in_seq=0 out_seq=0 2024-04-25T18:51:46.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.581+0000 7f56f7e00700 10 client.25841 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:46.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f56f7e00700 1 -- 172.21.15.63:0/3490178385 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x563a1b8d3820 con 0x563a1b9fb3e0 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f56f7e00700 10 client.25841 handle_command_reply: tid=0 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 client.25841 shutdown 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 msgr2=0x563a1b98c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f570409d770 tx=0x7f570400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 shutdown_connections 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a1b9fb3e0 0x563a1b9fd7d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a1b9844d0 0x563a1b9877c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a1b978d40 0x563a1b98c610 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 --2- 172.21.15.63:0/3490178385 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a1b979660 0x563a1b9870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 >> 172.21.15.63:0/3490178385 conn(0x563a1b8a8050 msgr2=0x563a1b97b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f56f7e00700 0 client.25841 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 shutdown_connections 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.587+0000 7f57239a3b80 1 -- 172.21.15.63:0/3490178385 wait complete. 2024-04-25T18:51:46.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.588+0000 7f57239a3b80 20 client.25841 trim_cache size 0 max 16384 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 31, 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 16, 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "9213c37": { 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 16, 2024-04-25T18:51:46.590 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 82.3, 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:46.591 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.588+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 msgr2=0x7f56f80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.588+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 0x7f56f80a6270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f570c005950 tx=0x7f570c009260 comp rx=0 tx=0).stop 2024-04-25T18:51:46.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.588+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 msgr2=0x7f57141a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.588+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f570400d010 tx=0x7f5704004d00 comp rx=0 tx=0).stop 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 shutdown_connections 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f56f80a3db0 0x7f56f80a6270 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5714113c10 0x7f571419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5714074c70 0x7f57141a5400 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 --2- 172.21.15.63:0/3042112751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5714075590 0x7f57141a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 >> 172.21.15.63:0/3042112751 conn(0x7f5714079ba0 msgr2=0x7f5714112a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 shutdown_connections 2024-04-25T18:51:46.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:46.589+0000 7f5719000700 1 -- 172.21.15.63:0/3042112751 wait complete. 2024-04-25T18:51:47.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 165432 5.09 MB/sec execute 106 sec latency 30877.335 ms 2024-04-25T18:51:47.712 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9213c37', seconds elapsed: 39.6 2024-04-25T18:51:47.712 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-04-25T18:51:47.712 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:51:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:48.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:48 smithi070 ceph-mon[32820]: pgmap v161: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 542 KiB/s wr, 39 op/s 2024-04-25T18:51:48.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 167034 5.16 MB/sec execute 107 sec latency 11.602 ms 2024-04-25T18:51:48.527 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:48 smithi063 ceph-mon[28006]: pgmap v161: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 542 KiB/s wr, 39 op/s 2024-04-25T18:51:48.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:48 smithi114 ceph-mon[32988]: pgmap v161: 97 pgs: 97 active+clean; 54 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 542 KiB/s wr, 39 op/s 2024-04-25T18:51:49.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:49 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:49.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 168305 5.17 MB/sec execute 108 sec latency 15.141 ms 2024-04-25T18:51:49.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:49 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:49.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:49 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:51:50.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:50 smithi063 ceph-mon[28006]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-04-25T18:51:50.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:50 smithi070 ceph-mon[32820]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-04-25T18:51:50.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 170272 5.23 MB/sec execute 109 sec latency 15.400 ms 2024-04-25T18:51:50.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:50 smithi114 ceph-mon[32988]: pgmap v162: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 1.3 MiB/s wr, 96 op/s 2024-04-25T18:51:50.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.883+0000 7f9e5d400700 1 -- 172.21.15.63:0/2672758970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 msgr2=0x7f9e58111b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.883+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2672758970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e58111b70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b0d0 tx=0x7f9e5000b3e0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 -- 172.21.15.63:0/2672758970 shutdown_connections 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2672758970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e58111b70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2672758970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58104ec0 0x7f9e581052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2672758970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58105870 0x7f9e581098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 -- 172.21.15.63:0/2672758970 >> 172.21.15.63:0/2672758970 conn(0x7f9e58100750 msgr2=0x7f9e58102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 -- 172.21.15.63:0/2672758970 shutdown_connections 2024-04-25T18:51:50.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.884+0000 7f9e5d400700 1 -- 172.21.15.63:0/2672758970 wait complete. 2024-04-25T18:51:50.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.885+0000 7f9e5d400700 1 Processor -- start 2024-04-25T18:51:50.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.885+0000 7f9e5d400700 1 -- start start 2024-04-25T18:51:50.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.885+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58104ec0 0x7f9e5819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 0x7f9e5819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e5819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e5d400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e58114e90 con 0x7f9e58104ec0 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e5d400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9e58114d10 con 0x7f9e58105870 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e5d400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9e58115010 con 0x7f9e58109fa0 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e55e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 0x7f9e5819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:50.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e56800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58104ec0 0x7f9e5819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e5819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e55e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 0x7f9e5819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38080/0 (socket says 172.21.15.63:38080) 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e55e00700 1 -- 172.21.15.63:0/2454323028 learned_addr learned my addr 172.21.15.63:0/2454323028 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 -- 172.21.15.63:0/2454323028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 msgr2=0x7f9e5819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 0x7f9e5819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 -- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58104ec0 msgr2=0x7f9e5819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58104ec0 0x7f9e5819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.886+0000 7f9e57200700 1 -- 172.21.15.63:0/2454323028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e581a02d0 con 0x7f9e58109fa0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e57200700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e5819fb90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b770 tx=0x7f9e50007980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5001f070 con 0x7f9e58109fa0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e50007dc0 con 0x7f9e58109fa0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5001b430 con 0x7f9e58109fa0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e581a0560 con 0x7f9e58109fa0 2024-04-25T18:51:50.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.887+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e581a7e40 con 0x7f9e58109fa0 2024-04-25T18:51:50.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.889+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e38005330 con 0x7f9e58109fa0 2024-04-25T18:51:50.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.892+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9e50022070 con 0x7f9e58109fa0 2024-04-25T18:51:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.894+0000 7f9e4f400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 0x7f9e3c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.894+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f9e5000f020 con 0x7f9e58109fa0 2024-04-25T18:51:50.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.894+0000 7f9e56800700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 0x7f9e3c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:50.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.895+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e50029020 con 0x7f9e58109fa0 2024-04-25T18:51:50.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:50.895+0000 7f9e56800700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 0x7f9e3c0a62f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009c60 tx=0x7f9e48009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:51.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.216+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e38005620 con 0x7f9e58109fa0 2024-04-25T18:51:51.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.217+0000 7f9e4f400700 1 -- 172.21.15.63:0/2454323028 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11872 (secure 0 0 0) 0x7f9e5009e070 con 0x7f9e58109fa0 2024-04-25T18:51:51.219 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:51:51.219 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:51:38.994954+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:51:51.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.220+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 msgr2=0x7f9e3c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:51.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.220+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 0x7f9e3c0a62f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009c60 tx=0x7f9e48009260 comp rx=0 tx=0).stop 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.220+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 msgr2=0x7f9e5819fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.220+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e5819fb90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b770 tx=0x7f9e50007980 comp rx=0 tx=0).stop 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 shutdown_connections 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e3c0a3e30 0x7f9e3c0a62f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9e58109fa0 0x7f9e5819fb90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e58104ec0 0x7f9e5819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:51.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 --2- 172.21.15.63:0/2454323028 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9e58105870 0x7f9e5819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:51.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 >> 172.21.15.63:0/2454323028 conn(0x7f9e58100750 msgr2=0x7f9e58101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:51.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 shutdown_connections 2024-04-25T18:51:51.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:51.221+0000 7f9e5d400700 1 -- 172.21.15.63:0/2454323028 wait complete. 2024-04-25T18:51:51.224 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 31 2024-04-25T18:51:51.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:51 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:51 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:51.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:51 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:51:51.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 171456 5.22 MB/sec execute 110 sec latency 13.452 ms 2024-04-25T18:51:51.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:51 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:51.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:51 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:51.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:51 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:51:51.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:51 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:51:51.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:51 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:51:51.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:51 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:51:52.226 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 82.35 --await-for 42.0' 2024-04-25T18:51:52.227 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 82.35 --await-for 42.0 2024-04-25T18:51:52.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:52 smithi070 ceph-mon[32820]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:52.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:52 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2454323028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:52.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:52.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 173261 5.27 MB/sec execute 111 sec latency 13.964 ms 2024-04-25T18:51:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:52 smithi114 ceph-mon[32988]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:52 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2454323028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:52 smithi063 ceph-mon[28006]: pgmap v163: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:52 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2454323028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:51:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:51:52.701 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 -- 172.21.15.63:0/2224705223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1881038f0 msgr2=0x7fe18810bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 --2- 172.21.15.63:0/2224705223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1881038f0 0x7fe18810bdd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe174009900 tx=0x7fe17402a490 comp rx=0 tx=0).stop 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 -- 172.21.15.63:0/2224705223 shutdown_connections 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 --2- 172.21.15.63:0/2224705223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1881038f0 0x7fe18810bdd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 --2- 172.21.15.63:0/2224705223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe18810c3a0 0x7fe18810e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 --2- 172.21.15.63:0/2224705223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe188102fd0 0x7fe1881033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 -- 172.21.15.63:0/2224705223 >> 172.21.15.63:0/2224705223 conn(0x7fe1880fc770 msgr2=0x7fe1880feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 -- 172.21.15.63:0/2224705223 shutdown_connections 2024-04-25T18:51:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.450+0000 7fe18e800640 1 -- 172.21.15.63:0/2224705223 wait complete. 2024-04-25T18:51:53.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.451+0000 7fe18e800640 1 Processor -- start 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 -- start start 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1881038f0 0x7fe1881a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18810c3a0 0x7fe18819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe188112030 con 0x7fe18810c3a0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe188111eb0 con 0x7fe188102fd0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18e800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881121b0 con 0x7fe1881038f0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38100/0 (socket says 172.21.15.63:38100) 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18d400640 1 -- 172.21.15.63:0/1030380721 learned_addr learned my addr 172.21.15.63:0/1030380721 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18ca00640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1881038f0 0x7fe1881a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.452+0000 7fe18de00640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18810c3a0 0x7fe18819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 -- 172.21.15.63:0/1030380721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1881038f0 msgr2=0x7fe1881a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1881038f0 0x7fe1881a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 -- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18810c3a0 msgr2=0x7fe18819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18810c3a0 0x7fe18819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 -- 172.21.15.63:0/1030380721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18819ad70 con 0x7fe188102fd0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18d400640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d950 tx=0x7fe17c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe186000640 1 -- 172.21.15.63:0/1030380721 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c017070 con 0x7fe188102fd0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18819b060 con 0x7fe188102fd0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe186000640 1 -- 172.21.15.63:0/1030380721 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe17c009c30 con 0x7fe188102fd0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe186000640 1 -- 172.21.15.63:0/1030380721 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c00f620 con 0x7fe188102fd0 2024-04-25T18:51:53.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.453+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1881a7300 con 0x7fe188102fd0 2024-04-25T18:51:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe186000640 1 -- 172.21.15.63:0/1030380721 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe17c010460 con 0x7fe188102fd0 2024-04-25T18:51:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe192b93740 1 Processor -- start 2024-04-25T18:51:53.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe192b93740 1 -- start start 2024-04-25T18:51:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe192b93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe192b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc701650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.457+0000 7fe192b93740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc703f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe192b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc65c900 con 0x55a5cc6f72d0 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe192b93740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc602ef0 con 0x55a5cc701b90 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe192b93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc6588c0 con 0x55a5cc6f69b0 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51658/0 (socket says 172.21.15.63:51658) 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc703f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 -- 172.21.15.63:0/1835613793 learned_addr learned my addr 172.21.15.63:0/1835613793 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18d400640 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc701650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 -- 172.21.15.63:0/1835613793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 msgr2=0x55a5cc703f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc703f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 -- 172.21.15.63:0/1835613793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 msgr2=0x55a5cc701650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc701650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.458+0000 7fe18de00640 1 -- 172.21.15.63:0/1835613793 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5cc36f210 con 0x55a5cc6f69b0 2024-04-25T18:51:53.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 174612 5.28 MB/sec execute 112 sec latency 17.976 ms 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe18de00640 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe180009f30 tx=0x7fe1800118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72be212e75f220e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe17a600640 1 -- 172.21.15.63:0/1835613793 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180004d20 con 0x55a5cc6f69b0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe186000640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 0x7fe1700a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe17a600640 1 -- 172.21.15.63:0/1835613793 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe180004050 con 0x55a5cc6f69b0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe17a600640 1 -- 172.21.15.63:0/1835613793 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180005240 con 0x55a5cc6f69b0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe186000640 1 -- 172.21.15.63:0/1030380721 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe17c0d0380 con 0x7fe188102fd0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.459+0000 7fe18d400640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 0x7fe1700a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- 172.21.15.63:0/1835613793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 msgr2=0x55a5cc6f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe180009f30 tx=0x7fe1800118d0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.466 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe18d400640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 0x7fe1700a6250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55a5cc6f7f50 tx=0x7fe17c000bc0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- 172.21.15.63:0/1835613793 shutdown_connections 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc703f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc701650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 --2- 172.21.15.63:0/1835613793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6f6d90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- 172.21.15.63:0/1835613793 >> 172.21.15.63:0/1835613793 conn(0x55a5cc6f2ec0 msgr2=0x55a5cc6f32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- 172.21.15.63:0/1835613793 shutdown_connections 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- 172.21.15.63:0/1835613793 wait complete. 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.460+0000 7fe192b93740 1 -- start start 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 Processor -- start 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6ff060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc701b90 0x55a5cc6ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.467 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc7047c0 con 0x55a5cc6f72d0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc616310 con 0x55a5cc6f69b0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a5cc704940 con 0x55a5cc701b90 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.461+0000 7fe192b93740 10 client.0 _ll_register_callbacks cb 0x55a5cc5e4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38124/0 (socket says 172.21.15.63:38124) 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 -- 172.21.15.63:0/3714731615 learned_addr learned my addr 172.21.15.63:0/3714731615 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc701b90 0x55a5cc6ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18ca00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6ff060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 -- 172.21.15.63:0/3714731615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc701b90 msgr2=0x55a5cc6ff5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc701b90 0x55a5cc6ff5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 -- 172.21.15.63:0/3714731615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 msgr2=0x55a5cc6ff060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6ff060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 -- 172.21.15.63:0/3714731615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5cc43b620 con 0x55a5cc6f69b0 2024-04-25T18:51:53.468 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe18de00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe180005b40 tx=0x7fe180005a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcfd73d853c744c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.462+0000 7fe16ea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18002c020 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe180028450 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe180004050 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe180037050 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe192b93740 10 client.25859 fetch_fsmap learned FSMap version 32 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a5cc710a20 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe180004050 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fe18000f640 con 0x55a5cc6f69b0 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.463+0000 7fe192b93740 10 client.25859 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe192b93740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe192b93740 20 client.25859 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc711e80 0x55a5cc714270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe192b93740 4 client.25859 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5cc6708b0 con 0x55a5cc711e80 2024-04-25T18:51:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe18ca00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc711e80 0x55a5cc714270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe18ca00640 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc711e80 0x55a5cc714270 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a78cac676ae58579 server_cookie=dddd6e99f19e3a95 in_seq=0 out_seq=0 2024-04-25T18:51:53.470 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.464+0000 7fe16ea00640 10 client.25859 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.465+0000 7fe16ea00640 1 -- 172.21.15.63:0/3714731615 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a5cc6708b0 con 0x55a5cc711e80 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.465+0000 7fe16ea00640 10 client.25859 handle_command_reply: tid=0 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.465+0000 7fe192b93740 1 client.25859 shutdown 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.465+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 msgr2=0x55a5cc6feb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.465+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe180005b40 tx=0x7fe180005a70 comp rx=0 tx=0).stop 2024-04-25T18:51:53.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 shutdown_connections 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc711e80 0x55a5cc714270 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc701b90 0x55a5cc6ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6ff060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 --2- 172.21.15.63:0/3714731615 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f69b0 0x55a5cc6feb20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe16ea00640 0 client.25859 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 >> 172.21.15.63:0/3714731615 conn(0x55a5cc6f2ec0 msgr2=0x55a5cc6f8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 shutdown_connections 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 1 -- 172.21.15.63:0/3714731615 wait complete. 2024-04-25T18:51:53.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.466+0000 7fe192b93740 20 client.25859 trim_cache size 0 max 16384 2024-04-25T18:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 Processor -- start 2024-04-25T18:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 -- start start 2024-04-25T18:51:53.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6f9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc7199b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881b25f0 con 0x55a5cc6f72d0 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881afe60 con 0x55a5cc701b90 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe192b93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881adcb0 con 0x55a5cc6f69b0 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe18de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.483+0000 7fe18de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51672/0 (socket says 172.21.15.63:51672) 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 -- 172.21.15.63:0/2546188295 learned_addr learned my addr 172.21.15.63:0/2546188295 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc7199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18ca00640 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 -- 172.21.15.63:0/2546188295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 msgr2=0x55a5cc7199b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc7199b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 -- 172.21.15.63:0/2546188295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 msgr2=0x55a5cc6f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 -- 172.21.15.63:0/2546188295 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5cc6f2970 con 0x55a5cc6f69b0 2024-04-25T18:51:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe18de00640 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe180009560 tx=0x7fe1800118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5868c9052d294d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.484+0000 7fe179c00640 1 -- 172.21.15.63:0/2546188295 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18002c070 con 0x55a5cc6f69b0 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe179c00640 1 -- 172.21.15.63:0/2546188295 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe180009a50 con 0x55a5cc6f69b0 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe179c00640 1 -- 172.21.15.63:0/2546188295 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1800288e0 con 0x55a5cc6f69b0 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 -- 172.21.15.63:0/2546188295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 msgr2=0x55a5cc70d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe180009560 tx=0x7fe1800118d0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 -- 172.21.15.63:0/2546188295 shutdown_connections 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc701b90 0x55a5cc7199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc6f72d0 0x55a5cc6f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 --2- 172.21.15.63:0/2546188295 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc70d680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.485+0000 7fe192b93740 1 -- 172.21.15.63:0/2546188295 >> 172.21.15.63:0/2546188295 conn(0x55a5cc6f2ec0 msgr2=0x55a5cc700be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.486+0000 7fe192b93740 1 -- 172.21.15.63:0/2546188295 shutdown_connections 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.486+0000 7fe192b93740 1 -- 172.21.15.63:0/2546188295 wait complete. 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.486+0000 7fe192b93740 1 -- start start 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.486+0000 7fe192b93740 1 Processor -- start 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc709d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc701b90 0x55a5cc707660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881ae320 con 0x55a5cc701b90 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881af790 con 0x55a5cc6f72d0 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe1881ae9f0 con 0x55a5cc6f69b0 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe192b93740 10 client.0 _ll_register_callbacks cb 0x7fe180004220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:53.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe18ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe18ca00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38134/0 (socket says 172.21.15.63:38134) 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe18ca00640 1 -- 172.21.15.63:0/4290482816 learned_addr learned my addr 172.21.15.63:0/4290482816 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe18de00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc709d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.487+0000 7fe18de00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc701b90 0x55a5cc707660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 -- 172.21.15.63:0/4290482816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 msgr2=0x55a5cc709d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc709d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 -- 172.21.15.63:0/4290482816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc701b90 msgr2=0x55a5cc707660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc701b90 0x55a5cc707660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 -- 172.21.15.63:0/4290482816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5cc707ca0 con 0x55a5cc6f72d0 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18de00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc709d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe18ca00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe17402aa50 tx=0x7fe174004340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21eefdac1ae4fdd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:53.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe16ea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174035040 con 0x55a5cc6f72d0 2024-04-25T18:51:53.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.488+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fe174005040 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe180009a50 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe174055050 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe192b93740 10 client.25865 fetch_fsmap learned FSMap version 32 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a5cc7808e0 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe180009a50 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fe174004770 con 0x55a5cc6f72d0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe192b93740 10 client.25865 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.489+0000 7fe192b93740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe192b93740 20 client.25865 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc781140 0x55a5cc783560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe192b93740 4 client.25865 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5cc66a320 con 0x55a5cc781140 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe18de00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc781140 0x55a5cc783560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:53.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe18de00640 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc781140 0x55a5cc783560 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f0d6552d4b9247f server_cookie=725309ea0a155927 in_seq=0 out_seq=0 2024-04-25T18:51:53.494 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.490+0000 7fe16ea00640 10 client.25865 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe16ea00640 1 -- 172.21.15.63:0/4290482816 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5cc66a320 con 0x55a5cc781140 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe16ea00640 10 client.25865 handle_command_reply: tid=0 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 client.25865 shutdown 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 msgr2=0x55a5cc707120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe17402aa50 tx=0x7fe174004340 comp rx=0 tx=0).stop 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 shutdown_connections 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a5cc781140 0x55a5cc783560 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a5cc6f72d0 0x55a5cc707120 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.536 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5cc701b90 0x55a5cc707660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 --2- 172.21.15.63:0/4290482816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a5cc6f69b0 0x55a5cc709d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 >> 172.21.15.63:0/4290482816 conn(0x55a5cc6f2ec0 msgr2=0x55a5cc6f1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe16ea00640 0 client.25865 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 shutdown_connections 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.534+0000 7fe192b93740 1 -- 172.21.15.63:0/4290482816 wait complete. 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.535+0000 7fe192b93740 20 client.25865 trim_cache size 0 max 16384 2024-04-25T18:51:53.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:51:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:51:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:51:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:51:53.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.535+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 msgr2=0x7fe1700a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.535+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 0x7fe1700a6250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x55a5cc6f7f50 tx=0x7fe17c000bc0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.535+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 msgr2=0x7fe1881a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.535+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d950 tx=0x7fe17c00de10 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 shutdown_connections 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe1700a3d90 0x7fe1700a6250 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe18810c3a0 0x7fe18819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe1881038f0 0x7fe1881a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 --2- 172.21.15.63:0/1030380721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe188102fd0 0x7fe1881a0440 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 >> 172.21.15.63:0/1030380721 conn(0x7fe1880fc770 msgr2=0x7fe188101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 shutdown_connections 2024-04-25T18:51:53.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:51:53.536+0000 7fe18e800640 1 -- 172.21.15.63:0/1030380721 wait complete. 2024-04-25T18:51:54.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:54 smithi070 ceph-mon[32820]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:54.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 176271 5.29 MB/sec execute 113 sec latency 35.624 ms 2024-04-25T18:51:54.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:54 smithi114 ceph-mon[32988]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:54.615 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:54 smithi063 ceph-mon[28006]: pgmap v164: 97 pgs: 97 active+clean; 55 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 961 KiB/s wr, 77 op/s 2024-04-25T18:51:55.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 -- 172.21.15.63:0/3738441393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a0075590 msgr2=0x7f41a010ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 --2- 172.21.15.63:0/3738441393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a0075590 0x7f41a010ef60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f418c009990 tx=0x7f418c009ca0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 -- 172.21.15.63:0/3738441393 shutdown_connections 2024-04-25T18:51:55.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 --2- 172.21.15.63:0/3738441393 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 --2- 172.21.15.63:0/3738441393 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a010f4a0 0x7f41a0111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 --2- 172.21.15.63:0/3738441393 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a0075590 0x7f41a010ef60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 -- 172.21.15.63:0/3738441393 >> 172.21.15.63:0/3738441393 conn(0x7f41a0079b10 msgr2=0x7f41a0079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.231+0000 7f41a7000700 1 -- 172.21.15.63:0/3738441393 shutdown_connections 2024-04-25T18:51:55.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.232+0000 7f41a7000700 1 -- 172.21.15.63:0/3738441393 wait complete. 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 Processor -- start 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 -- start start 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a0075590 0x7f41a019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 0x7f41a019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a0114e40 con 0x7f41a0075590 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41a0114cc0 con 0x7f41a010f4a0 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.233+0000 7f41a7000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41a0114fc0 con 0x7f41a0074c70 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f41a5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 0x7f41a019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51694/0 (socket says 172.21.15.63:51694) 2024-04-25T18:51:55.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 -- 172.21.15.63:0/472814186 learned_addr learned my addr 172.21.15.63:0/472814186 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419f400700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a0075590 0x7f41a019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 -- 172.21.15.63:0/472814186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 msgr2=0x7f41a019fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 0x7f41a019fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 -- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a0075590 msgr2=0x7f41a019b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a0075590 0x7f41a019b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f419fe00700 1 -- 172.21.15.63:0/472814186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a01a0280 con 0x7f41a0074c70 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.234+0000 7f41a5000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 0x7f41a019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f419fe00700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f419000e9a0 tx=0x7f419000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f419cc00700 1 -- 172.21.15.63:0/472814186 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419000ca10 con 0x7f41a0074c70 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f419cc00700 1 -- 172.21.15.63:0/472814186 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f419000cbb0 con 0x7f41a0074c70 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f419cc00700 1 -- 172.21.15.63:0/472814186 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190018680 con 0x7f41a0074c70 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a01a0570 con 0x7f41a0074c70 2024-04-25T18:51:55.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.235+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a01a7df0 con 0x7f41a0074c70 2024-04-25T18:51:55.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.238+0000 7f419cc00700 1 -- 172.21.15.63:0/472814186 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4190010430 con 0x7f41a0074c70 2024-04-25T18:51:55.241 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.240+0000 7f419cc00700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 0x7f41840a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.240+0000 7f419cc00700 1 -- 172.21.15.63:0/472814186 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f41900d09e0 con 0x7f41a0074c70 2024-04-25T18:51:55.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.240+0000 7f41b197fb80 1 Processor -- start 2024-04-25T18:51:55.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f419f400700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 0x7f41840a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 -- start start 2024-04-25T18:51:55.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c1395bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c1395be570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c139504020 con 0x55c1395b12d0 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1394aa8c0 con 0x55c1395bc180 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f41b197fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1394fffe0 con 0x55c1395b09b0 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51702/0 (socket says 172.21.15.63:51702) 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.241+0000 7f419fe00700 1 -- 172.21.15.63:0/2514974613 learned_addr learned my addr 172.21.15.63:0/2514974613 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f41a5000700 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c1395be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419f400700 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c1395bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 -- 172.21.15.63:0/2514974613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 msgr2=0x55c1395be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c1395be570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 -- 172.21.15.63:0/2514974613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 msgr2=0x55c1395bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c1395bbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 -- 172.21.15.63:0/2514974613 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1394a0810 con 0x55c1395b09b0 2024-04-25T18:51:55.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419f400700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 0x7f41840a6310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41a0110ff0 tx=0x7f418c00b260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.242+0000 7f419fe00700 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f41900992e0 tx=0x7f419009c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757a1c7571de1b36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.243+0000 7f4195600700 1 -- 172.21.15.63:0/2514974613 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41900a5580 con 0x55c1395b09b0 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.243+0000 7f4195600700 1 -- 172.21.15.63:0/2514974613 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4190085e50 con 0x55c1395b09b0 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.243+0000 7f4195600700 1 -- 172.21.15.63:0/2514974613 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41900a50f0 con 0x55c1395b09b0 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.243+0000 7f41b197fb80 1 -- 172.21.15.63:0/2514974613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 msgr2=0x55c1395b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.243+0000 7f41b197fb80 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f41900992e0 tx=0x7f419009c910 comp rx=0 tx=0).stop 2024-04-25T18:51:55.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 -- 172.21.15.63:0/2514974613 shutdown_connections 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395b0d90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c1395be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 --2- 172.21.15.63:0/2514974613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c1395bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 -- 172.21.15.63:0/2514974613 >> 172.21.15.63:0/2514974613 conn(0x55c1394df730 msgr2=0x55c1395ace60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 -- 172.21.15.63:0/2514974613 shutdown_connections 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.244+0000 7f41b197fb80 1 -- 172.21.15.63:0/2514974613 wait complete. 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 -- start start 2024-04-25T18:51:55.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 Processor -- start 2024-04-25T18:51:55.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b12d0 0x55c1395b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1395beca0 con 0x55c1395bc180 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1394bd9f0 con 0x55c1395b09b0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.245+0000 7f41b197fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1395bee20 con 0x55c1395b12d0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f41b197fb80 10 client.0 _ll_register_callbacks cb 0x55c13949c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38166/0 (socket says 172.21.15.63:38166) 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419f400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b12d0 0x55c1395b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 -- 172.21.15.63:0/980402430 learned_addr learned my addr 172.21.15.63:0/980402430 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f41a5000700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 -- 172.21.15.63:0/980402430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b12d0 msgr2=0x55c1395b8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b12d0 0x55c1395b8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 -- 172.21.15.63:0/980402430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 msgr2=0x55c1395b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 -- 172.21.15.63:0/980402430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c139451a60 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f419fe00700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f41900a5050 tx=0x7f41900ce060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282d6e7e5cad16ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.246+0000 7f418a000700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190066060 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f41900a50f0 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4190085e50 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41900a5580 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f41b197fb80 10 client.25877 fetch_fsmap learned FSMap version 32 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c13949ac80 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4190085e50 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.247+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f4190098660 con 0x55c1395b09b0 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 10 client.25877 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:51:55.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 10 client.25877 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 20 client.25877 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1395cb870 0x55c1395cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 4 client.25877 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c139511810 con 0x55c1395cb870 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f419f400700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1395cb870 0x55c1395cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f419f400700 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1395cb870 0x55c1395cdc60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6eb460684dafe52 server_cookie=51802401a1788721 in_seq=0 out_seq=0 2024-04-25T18:51:55.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.248+0000 7f418a000700 10 client.25877 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.249+0000 7f418a000700 1 -- 172.21.15.63:0/980402430 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c139511810 con 0x55c1395cb870 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.249+0000 7f418a000700 10 client.25877 handle_command_reply: tid=0 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.249+0000 7f41b197fb80 1 client.25877 shutdown 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.249+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 msgr2=0x55c1395bb920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.249+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f41900a5050 tx=0x7f41900ce060 comp rx=0 tx=0).stop 2024-04-25T18:51:55.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 shutdown_connections 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1395cb870 0x55c1395cdc60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b09b0 0x55c1395bb920 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 --2- 172.21.15.63:0/980402430 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b12d0 0x55c1395b8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f418a000700 0 client.25877 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 >> 172.21.15.63:0/980402430 conn(0x55c1394df730 msgr2=0x55c1395aca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 shutdown_connections 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 1 -- 172.21.15.63:0/980402430 wait complete. 2024-04-25T18:51:55.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.250+0000 7f41b197fb80 20 client.25877 trim_cache size 0 max 16384 2024-04-25T18:51:55.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.274+0000 7f41b197fb80 1 Processor -- start 2024-04-25T18:51:55.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.274+0000 7f41b197fb80 1 -- start start 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b12d0 0x55c1395d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01b1e60 con 0x55c1395bc180 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01af690 con 0x55c1395b12d0 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41b197fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01ad4e0 con 0x55c1395b09b0 2024-04-25T18:51:55.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51722/0 (socket says 172.21.15.63:51722) 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f419fe00700 1 -- 172.21.15.63:0/1051693334 learned_addr learned my addr 172.21.15.63:0/1051693334 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f419f400700 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b12d0 0x55c1395d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.275+0000 7f41a5000700 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 -- 172.21.15.63:0/1051693334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b12d0 msgr2=0x55c1395d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b12d0 0x55c1395d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 -- 172.21.15.63:0/1051693334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 msgr2=0x55c1395b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 -- 172.21.15.63:0/1051693334 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1395b24f0 con 0x55c1395b09b0 2024-04-25T18:51:55.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f419fe00700 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f419009c850 tx=0x7f419009d0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575c902e9e8ad926 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f4194c00700 1 -- 172.21.15.63:0/1051693334 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190066070 con 0x55c1395b09b0 2024-04-25T18:51:55.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f4194c00700 1 -- 172.21.15.63:0/1051693334 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4190085e50 con 0x55c1395b09b0 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.276+0000 7f4194c00700 1 -- 172.21.15.63:0/1051693334 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41900a50f0 con 0x55c1395b09b0 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 -- 172.21.15.63:0/1051693334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 msgr2=0x55c1395d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f419009c850 tx=0x7f419009d0c0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 -- 172.21.15.63:0/1051693334 shutdown_connections 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c1395d0c90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395bc180 0x55c1395b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 --2- 172.21.15.63:0/1051693334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395b12d0 0x55c1395d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 -- 172.21.15.63:0/1051693334 >> 172.21.15.63:0/1051693334 conn(0x55c1394df730 msgr2=0x55c1395c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.277+0000 7f41b197fb80 1 -- 172.21.15.63:0/1051693334 shutdown_connections 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.278+0000 7f41b197fb80 1 -- 172.21.15.63:0/1051693334 wait complete. 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.278+0000 7f41b197fb80 1 -- start start 2024-04-25T18:51:55.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.278+0000 7f41b197fb80 1 Processor -- start 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c13962b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c13962bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01adb50 con 0x55c1395b12d0 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01aefc0 con 0x55c1395bc180 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01ae220 con 0x55c1395b09b0 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41b197fb80 10 client.0 _ll_register_callbacks cb 0x55c13949c8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41a5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41a5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38174/0 (socket says 172.21.15.63:38174) 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f41a5000700 1 -- 172.21.15.63:0/4099469367 learned_addr learned my addr 172.21.15.63:0/4099469367 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:51:55.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f419fe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c13962b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.279+0000 7f419f400700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c13962bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 -- 172.21.15.63:0/4099469367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 msgr2=0x55c13962b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c13962b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 -- 172.21.15.63:0/4099469367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 msgr2=0x55c13962bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c13962bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 -- 172.21.15.63:0/4099469367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1396264e0 con 0x55c1395bc180 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41a5000700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4198004730 tx=0x7f4198004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=260ef5b1a80d8425 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f418a000700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980048f0 con 0x55c1395bc180 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f4198004db0 con 0x55c1395bc180 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.280+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4190085e50 con 0x55c1395bc180 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4198017a90 con 0x55c1395bc180 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 10 client.25883 fetch_fsmap learned FSMap version 32 2024-04-25T18:51:55.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c139633230 con 0x55c1395bc180 2024-04-25T18:51:55.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4190085e50 con 0x55c1395bc180 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f419801e070 con 0x55c1395bc180 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 10 client.25883 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 10 client.25883 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 20 client.25883 populate_metadata read hostname 'smithi063' 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c139633a00 0x55c139635e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 4 client.25883 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.35, "await_for": 42.0}] 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.281+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 82.35, "await_for": 42.0}) -- 0x55c13950b8b0 con 0x55c139633a00 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.282+0000 7f419fe00700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c139633a00 0x55c139635e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.282+0000 7f419fe00700 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c139633a00 0x55c139635e20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d55de010b6cee344 server_cookie=b9e66f36a78816c7 in_seq=0 out_seq=0 2024-04-25T18:51:55.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.282+0000 7f418a000700 10 client.25883 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:55.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.350+0000 7f418a000700 1 -- 172.21.15.63:0/4099469367 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c13950b8b0 con 0x55c139633a00 2024-04-25T18:51:55.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.350+0000 7f418a000700 10 client.25883 handle_command_reply: tid=0 2024-04-25T18:51:55.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.350+0000 7f41b197fb80 1 client.25883 shutdown 2024-04-25T18:51:55.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.350+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 msgr2=0x55c139625de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.350+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4198004730 tx=0x7f4198004760 comp rx=0 tx=0).stop 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 shutdown_connections 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c139633a00 0x55c139635e20 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1395b09b0 0x55c13962b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1395bc180 0x55c139625de0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 --2- 172.21.15.63:0/4099469367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1395b12d0 0x55c13962bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 >> 172.21.15.63:0/4099469367 conn(0x55c1394df730 msgr2=0x55c1395abf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f418a000700 0 client.25883 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 shutdown_connections 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 1 -- 172.21.15.63:0/4099469367 wait complete. 2024-04-25T18:51:55.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.351+0000 7f41b197fb80 20 client.25883 trim_cache size 0 max 16384 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 32, 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 18, 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "bd1a99b1": { 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 18, 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:51:55.354 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 82.3, 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:55.355 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:51:55.356 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 msgr2=0x7f41840a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 0x7f41840a6310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f41a0110ff0 tx=0x7f418c00b260 comp rx=0 tx=0).stop 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 msgr2=0x7f41a019b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f419000e9a0 tx=0x7f419000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 shutdown_connections 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f41840a3e50 0x7f41840a6310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f41a0074c70 0x7f41a019b270 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41a010f4a0 0x7f41a019fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 --2- 172.21.15.63:0/472814186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a0075590 0x7f41a019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:51:55.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 >> 172.21.15.63:0/472814186 conn(0x7f41a0079b10 msgr2=0x7f41a00ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:51:55.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 shutdown_connections 2024-04-25T18:51:55.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:51:55.353+0000 7f41a7000700 1 -- 172.21.15.63:0/472814186 wait complete. 2024-04-25T18:51:55.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.34 MB/sec execute 114 sec latency 164.410 ms 2024-04-25T18:51:56.428 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'bd1a99b1', quiesce took 8.7 seconds. Will release after: 22.35 2024-04-25T18:51:56.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:56 smithi070 ceph-mon[32820]: pgmap v165: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-25T18:51:56.459 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.29 MB/sec execute 115 sec latency 1164.542 ms 2024-04-25T18:51:56.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:56 smithi114 ceph-mon[32988]: pgmap v165: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-25T18:51:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:56 smithi063 ceph-mon[28006]: pgmap v165: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-04-25T18:51:57.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.25 MB/sec execute 116 sec latency 2164.639 ms 2024-04-25T18:51:58.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:51:58 smithi070 ceph-mon[32820]: pgmap v166: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-25T18:51:58.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.20 MB/sec execute 117 sec latency 3164.806 ms 2024-04-25T18:51:58.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:51:58 smithi114 ceph-mon[32988]: pgmap v166: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-25T18:51:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:51:58 smithi063 ceph-mon[28006]: pgmap v166: 97 pgs: 97 active+clean; 57 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-25T18:51:59.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.16 MB/sec execute 118 sec latency 4164.964 ms 2024-04-25T18:52:00.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.12 MB/sec execute 119 sec latency 5165.113 ms 2024-04-25T18:52:00.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:00 smithi114 ceph-mon[32988]: pgmap v167: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-25T18:52:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:00 smithi063 ceph-mon[28006]: pgmap v167: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-25T18:52:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:00 smithi070 ceph-mon[32820]: pgmap v167: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.6 MiB/s wr, 195 op/s 2024-04-25T18:52:01.460 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.07 MB/sec execute 120 sec latency 6165.256 ms 2024-04-25T18:52:02.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 5.03 MB/sec execute 121 sec latency 7165.414 ms 2024-04-25T18:52:02.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:02 smithi114 ceph-mon[32988]: pgmap v168: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:02.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:02 smithi063 ceph-mon[28006]: pgmap v168: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:02 smithi070 ceph-mon[32820]: pgmap v168: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:03.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.99 MB/sec execute 122 sec latency 8165.578 ms 2024-04-25T18:52:04.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.95 MB/sec execute 123 sec latency 9165.674 ms 2024-04-25T18:52:04.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:04 smithi114 ceph-mon[32988]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:04.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:04 smithi063 ceph-mon[28006]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:04 smithi070 ceph-mon[32820]: pgmap v169: 97 pgs: 97 active+clean; 57 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 880 KiB/s wr, 138 op/s 2024-04-25T18:52:05.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.91 MB/sec execute 124 sec latency 10165.823 ms 2024-04-25T18:52:06.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.87 MB/sec execute 125 sec latency 11165.923 ms 2024-04-25T18:52:06.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:06 smithi114 ceph-mon[32988]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 KiB/s wr, 157 op/s 2024-04-25T18:52:06.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:06 smithi063 ceph-mon[28006]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 KiB/s wr, 157 op/s 2024-04-25T18:52:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.683+0000 7fdff6200640 1 -- 172.21.15.63:0/3844960172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 msgr2=0x7fdff0110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.683+0000 7fdff6200640 1 --2- 172.21.15.63:0/3844960172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff0110ac0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdfe8009ca0 tx=0x7fdfe802a8c0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 -- 172.21.15.63:0/3844960172 shutdown_connections 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 --2- 172.21.15.63:0/3844960172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff0110ac0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 --2- 172.21.15.63:0/3844960172 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 --2- 172.21.15.63:0/3844960172 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 -- 172.21.15.63:0/3844960172 >> 172.21.15.63:0/3844960172 conn(0x7fdff00fc770 msgr2=0x7fdff00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 -- 172.21.15.63:0/3844960172 shutdown_connections 2024-04-25T18:52:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.684+0000 7fdff6200640 1 -- 172.21.15.63:0/3844960172 wait complete. 2024-04-25T18:52:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 Processor -- start 2024-04-25T18:52:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 -- start start 2024-04-25T18:52:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff01a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdff0114220 con 0x7fdff010e6d0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01140a0 con 0x7fdff0100b20 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.685+0000 7fdff6200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01143a0 con 0x7fdff01014d0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33550/0 (socket says 172.21.15.63:33550) 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdfe7e00640 1 -- 172.21.15.63:0/1124994761 learned_addr learned my addr 172.21.15.63:0/1124994761 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff4e00640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff01a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 -- 172.21.15.63:0/1124994761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 msgr2=0x7fdff01a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 -- 172.21.15.63:0/1124994761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 msgr2=0x7fdff01a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff01a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff5800640 1 -- 172.21.15.63:0/1124994761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff019ad80 con 0x7fdff010e6d0 2024-04-25T18:52:06.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.686+0000 7fdff4e00640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff01a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdff5800640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff019a600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802aeb0 tx=0x7fdfe80317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdfee000640 1 -- 172.21.15.63:0/1124994761 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe8003b70 con 0x7fdff010e6d0 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdfee000640 1 -- 172.21.15.63:0/1124994761 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe8003d10 con 0x7fdff010e6d0 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff019b010 con 0x7fdff010e6d0 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdfee000640 1 -- 172.21.15.63:0/1124994761 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe802f970 con 0x7fdff010e6d0 2024-04-25T18:52:06.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.687+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff01a72c0 con 0x7fdff010e6d0 2024-04-25T18:52:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.692+0000 7fdffa5ea740 1 Processor -- start 2024-04-25T18:52:06.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.692+0000 7fdfee000640 1 -- 172.21.15.63:0/1124994761 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdfe802fb10 con 0x7fdff010e6d0 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.692+0000 7fdffa5ea740 1 -- start start 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 0x55a4edbda650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbdcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a4edb35900 con 0x55a4edbd02d0 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a4edadbef0 con 0x55a4edbdab90 2024-04-25T18:52:06.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a4edb318c0 con 0x55a4edbcf9b0 2024-04-25T18:52:06.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdff4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 0x55a4edbda650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdfe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33556/0 (socket says 172.21.15.63:33556) 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdfe7e00640 1 -- 172.21.15.63:0/3961891266 learned_addr learned my addr 172.21.15.63:0/3961891266 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.693+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbdcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 -- 172.21.15.63:0/3961891266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 msgr2=0x55a4edbcfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 -- 172.21.15.63:0/3961891266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 msgr2=0x55a4edbdcf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbdcf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 -- 172.21.15.63:0/3961891266 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4ed848210 con 0x55a4edbd02d0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfee000640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 0x7fdfd40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdff4e00640 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 0x55a4edbda650 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009fd0 tx=0x7fdfe00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c15614b0f21289 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfee000640 1 -- 172.21.15.63:0/1124994761 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fdfe80f1a80 con 0x7fdff010e6d0 2024-04-25T18:52:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 0x7fdfd40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfe6000640 1 -- 172.21.15.63:0/3961891266 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0004030 con 0x55a4edbd02d0 2024-04-25T18:52:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfe6000640 1 -- 172.21.15.63:0/3961891266 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe00054d0 con 0x55a4edbd02d0 2024-04-25T18:52:06.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.694+0000 7fdfe6000640 1 -- 172.21.15.63:0/3961891266 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe0004bc0 con 0x55a4edbd02d0 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 0x7fdfd40a61d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55a4edbcd2e0 tx=0x7fdfd8005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 -- 172.21.15.63:0/3961891266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 msgr2=0x55a4edbda650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 0x55a4edbda650 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009fd0 tx=0x7fdfe00118d0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 -- 172.21.15.63:0/3961891266 shutdown_connections 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbdcf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbd02d0 0x55a4edbda650 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/3961891266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 -- 172.21.15.63:0/3961891266 >> 172.21.15.63:0/3961891266 conn(0x55a4edbcbec0 msgr2=0x55a4edbcc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.695+0000 7fdffa5ea740 1 -- 172.21.15.63:0/3961891266 shutdown_connections 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.696+0000 7fdffa5ea740 1 -- 172.21.15.63:0/3961891266 wait complete. 2024-04-25T18:52:06.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.696+0000 7fdffa5ea740 1 -- start start 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.696+0000 7fdffa5ea740 1 Processor -- start 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbd7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbd02d0 0x55a4edbd8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a4edbdd7c0 con 0x55a4edbcf9b0 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a4edaef310 con 0x55a4edbd02d0 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a4edbdd940 con 0x55a4edbdab90 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdffa5ea740 10 client.0 _ll_register_callbacks cb 0x55a4edabdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdff4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbd7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdff5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdff5800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33566/0 (socket says 172.21.15.63:33566) 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdff5800640 1 -- 172.21.15.63:0/1079702208 learned_addr learned my addr 172.21.15.63:0/1079702208 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbd02d0 0x55a4edbd8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.697+0000 7fdff5800640 1 -- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbd02d0 msgr2=0x55a4edbd8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdff5800640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbd02d0 0x55a4edbd8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdff5800640 1 -- 172.21.15.63:0/1079702208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 msgr2=0x55a4edbd7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdff5800640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbd7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdff5800640 1 -- 172.21.15.63:0/1079702208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4ed914620 con 0x55a4edbdab90 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdff5800640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80bd7c0 tx=0x7fdfe80bd420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ae3762e6b3ab4a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdfdea00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:52:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe80c1e70 con 0x55a4edbdab90 2024-04-25T18:52:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fdfe80ef3e0 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfe00054d0 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdfe80c6730 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdffa5ea740 10 client.25909 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.698+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4edbdd7c0 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe00054d0 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fdfe80c62a0 con 0x55a4edbdab90 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 10 client.25909 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 10 client.25909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 20 client.25909 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edbeafd0 0x55a4edbed410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 4 client.25909 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4edb498b0 con 0x55a4edbeafd0 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.699+0000 7fdff4e00640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edbeafd0 0x55a4edbed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.700+0000 7fdff4e00640 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edbeafd0 0x55a4edbed410 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85a147ca289518c server_cookie=b9dd48c37635deda in_seq=0 out_seq=0 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.700+0000 7fdfdea00640 10 client.25909 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdfdea00640 1 -- 172.21.15.63:0/1079702208 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4edb498b0 con 0x55a4edbeafd0 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdfdea00640 10 client.25909 handle_command_reply: tid=0 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 client.25909 shutdown 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 msgr2=0x55a4edbd85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdfe80bd7c0 tx=0x7fdfe80bd420 comp rx=0 tx=0).stop 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 shutdown_connections 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edbeafd0 0x55a4edbed410 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbd02d0 0x55a4edbd8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbdab90 0x55a4edbd85d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/1079702208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbd7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdfdea00640 0 client.25909 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 >> 172.21.15.63:0/1079702208 conn(0x55a4edbcbec0 msgr2=0x55a4edbd1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 shutdown_connections 2024-04-25T18:52:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 1 -- 172.21.15.63:0/1079702208 wait complete. 2024-04-25T18:52:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.701+0000 7fdffa5ea740 20 client.25909 trim_cache size 0 max 16384 2024-04-25T18:52:06.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:06 smithi070 ceph-mon[32820]: pgmap v170: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 KiB/s wr, 157 op/s 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 Processor -- start 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 -- start start 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbd2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01b2410 con 0x55a4edbdab90 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01afc80 con 0x55a4edbcf9b0 2024-04-25T18:52:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.718+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01adad0 con 0x55a4edbd02d0 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43602/0 (socket says 172.21.15.63:43602) 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 -- 172.21.15.63:0/2516967389 learned_addr learned my addr 172.21.15.63:0/2516967389 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff4e00640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbd2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 -- 172.21.15.63:0/2516967389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 msgr2=0x55a4edbd2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbd2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 -- 172.21.15.63:0/2516967389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 msgr2=0x55a4edbcb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff5800640 1 -- 172.21.15.63:0/2516967389 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4edbc7820 con 0x55a4edbdab90 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.719+0000 7fdff4e00640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdff5800640 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802ade0 tx=0x7fdfe80bd550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85ab141b2eecd5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdfe5600640 1 -- 172.21.15.63:0/2516967389 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe80be1c0 con 0x55a4edbdab90 2024-04-25T18:52:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdfe5600640 1 -- 172.21.15.63:0/2516967389 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe8036040 con 0x55a4edbdab90 2024-04-25T18:52:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdfe5600640 1 -- 172.21.15.63:0/2516967389 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe80c62a0 con 0x55a4edbdab90 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdffa5ea740 1 -- 172.21.15.63:0/2516967389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 msgr2=0x55a4edbf2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.720+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802ade0 tx=0x7fdfe80bd550 comp rx=0 tx=0).stop 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 -- 172.21.15.63:0/2516967389 shutdown_connections 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbdab90 0x55a4edbf2d20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbd2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/2516967389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbcf9b0 0x55a4edbcb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 -- 172.21.15.63:0/2516967389 >> 172.21.15.63:0/2516967389 conn(0x55a4edbcbec0 msgr2=0x55a4edbd83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 -- 172.21.15.63:0/2516967389 shutdown_connections 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 -- 172.21.15.63:0/2516967389 wait complete. 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 -- start start 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.721+0000 7fdffa5ea740 1 Processor -- start 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbe03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbe0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01ae140 con 0x55a4edbcf9b0 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01af5b0 con 0x55a4edbdab90 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdff01ae810 con 0x55a4edbd02d0 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdffa5ea740 10 client.0 _ll_register_callbacks cb 0x7fdfe80b9e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdff4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdff4e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43616/0 (socket says 172.21.15.63:43616) 2024-04-25T18:52:06.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdff4e00640 1 -- 172.21.15.63:0/4026304120 learned_addr learned my addr 172.21.15.63:0/4026304120 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbe03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.722+0000 7fdff5800640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbe0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 -- 172.21.15.63:0/4026304120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 msgr2=0x55a4edbe03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbe03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 -- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 msgr2=0x55a4edbe0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbe0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 -- 172.21.15.63:0/4026304120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4edbe1170 con 0x55a4edbcf9b0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdff4e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00057c0 tx=0x7fdfe00045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ce0a6e84e74437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdfdea00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe002bd60 con 0x55a4edbcf9b0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fdfe002b240 con 0x55a4edbcf9b0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.723+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfe8036040 con 0x55a4edbcf9b0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdfe003a050 con 0x55a4edbcf9b0 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 10 client.15717 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:06.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4edc51680 con 0x55a4edbcf9b0 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfe8036040 con 0x55a4edbcf9b0 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fdfe000f940 con 0x55a4edbcf9b0 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 10 client.15717 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 10 client.15717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 20 client.15717 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edc51c70 0x55a4edc54090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 4 client.15717 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:52:06.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.724+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a4edb43320 con 0x55a4edc51c70 2024-04-25T18:52:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.725+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edc51c70 0x55a4edc54090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.725+0000 7fdfe7e00640 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edc51c70 0x55a4edc54090 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815931838820eabc server_cookie=1c85e70bc3f04c61 in_seq=0 out_seq=0 2024-04-25T18:52:06.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.725+0000 7fdfdea00640 10 client.15717 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.751+0000 7fdfdea00640 1 -- 172.21.15.63:0/4026304120 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a4edb43320 con 0x55a4edc51c70 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.751+0000 7fdfdea00640 10 client.15717 handle_command_reply: tid=0 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.751+0000 7fdffa5ea740 1 client.15717 shutdown 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 msgr2=0x55a4edbe3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00057c0 tx=0x7fdfe00045a0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 shutdown_connections 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a4edc51c70 0x55a4edc54090 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a4edbdab90 0x55a4edbe0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a4edbd02d0 0x55a4edbe03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 --2- 172.21.15.63:0/4026304120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a4edbcf9b0 0x55a4edbe3010 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdfdea00640 0 client.15717 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 >> 172.21.15.63:0/4026304120 conn(0x55a4edbcbec0 msgr2=0x55a4edbd9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 shutdown_connections 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 1 -- 172.21.15.63:0/4026304120 wait complete. 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.752+0000 7fdffa5ea740 20 client.15717 trim_cache size 0 max 16384 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:52:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:52:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:52:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:52:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 msgr2=0x7fdfd40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 0x7fdfd40a61d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x55a4edbcd2e0 tx=0x7fdfd8005f90 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 msgr2=0x7fdff019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff019a600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdfe802aeb0 tx=0x7fdfe80317c0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 shutdown_connections 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdfd40a3d10 0x7fdfd40a61d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdff010e6d0 0x7fdff019a600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdff01014d0 0x7fdff01a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 --2- 172.21.15.63:0/1124994761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdff0100b20 0x7fdff01a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 >> 172.21.15.63:0/1124994761 conn(0x7fdff00fc770 msgr2=0x7fdff00feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 shutdown_connections 2024-04-25T18:52:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:06.753+0000 7fdff6200640 1 -- 172.21.15.63:0/1124994761 wait complete. 2024-04-25T18:52:07.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.83 MB/sec execute 126 sec latency 12166.073 ms 2024-04-25T18:52:08.461 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.79 MB/sec execute 127 sec latency 13166.176 ms 2024-04-25T18:52:08.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:08 smithi114 ceph-mon[32988]: pgmap v171: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 154 KiB/s wr, 52 op/s 2024-04-25T18:52:08.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:08 smithi063 ceph-mon[28006]: pgmap v171: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 154 KiB/s wr, 52 op/s 2024-04-25T18:52:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:08 smithi070 ceph-mon[32820]: pgmap v171: 97 pgs: 97 active+clean; 57 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 154 KiB/s wr, 52 op/s 2024-04-25T18:52:09.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.76 MB/sec execute 128 sec latency 14166.315 ms 2024-04-25T18:52:10.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.72 MB/sec execute 129 sec latency 15166.441 ms 2024-04-25T18:52:10.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:10 smithi114 ceph-mon[32988]: pgmap v172: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 168 KiB/s wr, 63 op/s 2024-04-25T18:52:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:10 smithi063 ceph-mon[28006]: pgmap v172: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 168 KiB/s wr, 63 op/s 2024-04-25T18:52:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:10 smithi070 ceph-mon[32820]: pgmap v172: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 168 KiB/s wr, 63 op/s 2024-04-25T18:52:11.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.68 MB/sec execute 130 sec latency 16166.563 ms 2024-04-25T18:52:12.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.65 MB/sec execute 131 sec latency 17166.730 ms 2024-04-25T18:52:12.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:12 smithi114 ceph-mon[32988]: pgmap v173: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:12.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:12 smithi063 ceph-mon[28006]: pgmap v173: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:12 smithi070 ceph-mon[32820]: pgmap v173: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:13.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.61 MB/sec execute 132 sec latency 18166.876 ms 2024-04-25T18:52:14.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.58 MB/sec execute 133 sec latency 19167.053 ms 2024-04-25T18:52:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:14 smithi114 ceph-mon[32988]: pgmap v174: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:14 smithi063 ceph-mon[28006]: pgmap v174: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:14 smithi070 ceph-mon[32820]: pgmap v174: 97 pgs: 97 active+clean; 54 MiB data, 866 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 48 KiB/s wr, 30 op/s 2024-04-25T18:52:15.462 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.54 MB/sec execute 134 sec latency 20167.153 ms 2024-04-25T18:52:16.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.51 MB/sec execute 135 sec latency 21167.302 ms 2024-04-25T18:52:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:16 smithi114 ceph-mon[32988]: pgmap v175: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.0 MiB/s wr, 62 op/s 2024-04-25T18:52:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:16 smithi063 ceph-mon[28006]: pgmap v175: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.0 MiB/s wr, 62 op/s 2024-04-25T18:52:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:16 smithi070 ceph-mon[32820]: pgmap v175: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.0 MiB/s wr, 62 op/s 2024-04-25T18:52:17.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.48 MB/sec execute 136 sec latency 22167.453 ms 2024-04-25T18:52:18.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.44 MB/sec execute 137 sec latency 23167.601 ms 2024-04-25T18:52:18.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:18 smithi114 ceph-mon[32988]: pgmap v176: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:18 smithi063 ceph-mon[28006]: pgmap v176: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:18 smithi070 ceph-mon[32820]: pgmap v176: 97 pgs: 97 active+clean; 58 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:18.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:52:19.254 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:52:19.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.41 MB/sec execute 138 sec latency 24167.692 ms 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 -- 172.21.15.63:0/3630167874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103b40 msgr2=0x7f729810c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 --2- 172.21.15.63:0/3630167874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103b40 0x7f729810c020 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7284009900 tx=0x7f728402a480 comp rx=0 tx=0).stop 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 -- 172.21.15.63:0/3630167874 shutdown_connections 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 --2- 172.21.15.63:0/3630167874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 0x7f729810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 --2- 172.21.15.63:0/3630167874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103b40 0x7f729810c020 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 --2- 172.21.15.63:0/3630167874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103220 0x7f7298103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 -- 172.21.15.63:0/3630167874 >> 172.21.15.63:0/3630167874 conn(0x7f7298077fd0 msgr2=0x7f72980783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 -- 172.21.15.63:0/3630167874 shutdown_connections 2024-04-25T18:52:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.901+0000 7f72a0200640 1 -- 172.21.15.63:0/3630167874 wait complete. 2024-04-25T18:52:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.902+0000 7f72a0200640 1 Processor -- start 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.902+0000 7f72a0200640 1 -- start start 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103b40 0x7f729819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 0x7f72981985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7298112200 con 0x7f729810c560 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7298112080 con 0x7f7298103b40 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f72a0200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7298112380 con 0x7f7298103220 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:53768/0 (socket says 172.21.15.63:53768) 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729e200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 0x7f72981985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729d800640 1 -- 172.21.15.63:0/71742055 learned_addr learned my addr 172.21.15.63:0/71742055 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729ce00640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103b40 0x7f729819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.903+0000 7f729d800640 1 -- 172.21.15.63:0/71742055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103b40 msgr2=0x7f729819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f729d800640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103b40 0x7f729819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f729d800640 1 -- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 msgr2=0x7f72981985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f729d800640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 0x7f72981985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f729d800640 1 -- 172.21.15.63:0/71742055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7298198d20 con 0x7f7298103220 2024-04-25T18:52:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f729d800640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9f0 tx=0x7f728c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f7296000640 1 -- 172.21.15.63:0/71742055 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c00ccc0 con 0x7f7298103220 2024-04-25T18:52:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7298199010 con 0x7f7298103220 2024-04-25T18:52:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f7296000640 1 -- 172.21.15.63:0/71742055 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f728c00ce60 con 0x7f7298103220 2024-04-25T18:52:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f7296000640 1 -- 172.21.15.63:0/71742055 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c010430 con 0x7f7298103220 2024-04-25T18:52:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.904+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72981a52b0 con 0x7f7298103220 2024-04-25T18:52:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.907+0000 7f72a3172740 1 Processor -- start 2024-04-25T18:52:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 -- start start 2024-04-25T18:52:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19c8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19c8550 0x559ac19d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f7296000640 1 -- 172.21.15.63:0/71742055 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f728c0046c0 con 0x7f7298103220 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559ac1929900 con 0x559ac19d2e10 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559ac18cfef0 con 0x559ac19c7c30 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f72a3172740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559ac19258c0 con 0x559ac19c8550 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f729ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f729d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19c8550 0x559ac19d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f729ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45888/0 (socket says 172.21.15.63:45888) 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f729ce00640 1 -- 172.21.15.63:0/1487691063 learned_addr learned my addr 172.21.15.63:0/1487691063 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.908+0000 7f729e200640 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 -- 172.21.15.63:0/1487691063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19c8550 msgr2=0x559ac19d28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19c8550 0x559ac19d28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 -- 172.21.15.63:0/1487691063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 msgr2=0x559ac19c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.912 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 -- 172.21.15.63:0/1487691063 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac163c210 con 0x559ac19d2e10 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.909+0000 7f729ce00640 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7284000c00 tx=0x7f7284004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8369cfbca1e5d454 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f728aa00640 1 -- 172.21.15.63:0/1487691063 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284035040 con 0x559ac19d2e10 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f7296000640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 0x7f72800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f728aa00640 1 -- 172.21.15.63:0/1487691063 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72840054f0 con 0x559ac19d2e10 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f728aa00640 1 -- 172.21.15.63:0/1487691063 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72840035f0 con 0x559ac19d2e10 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f7296000640 1 -- 172.21.15.63:0/71742055 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f728c0d0bc0 con 0x7f7298103220 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.910+0000 7f729e200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 0x7f72800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- 172.21.15.63:0/1487691063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 msgr2=0x559ac19d5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7284000c00 tx=0x7f7284004040 comp rx=0 tx=0).stop 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- 172.21.15.63:0/1487691063 shutdown_connections 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19d2e10 0x559ac19d5200 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19c8550 0x559ac19d28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 --2- 172.21.15.63:0/1487691063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- 172.21.15.63:0/1487691063 >> 172.21.15.63:0/1487691063 conn(0x559ac19c4140 msgr2=0x559ac19c4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- 172.21.15.63:0/1487691063 shutdown_connections 2024-04-25T18:52:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f729e200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 0x7f72800a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7290005fd0 tx=0x7f7290009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- 172.21.15.63:0/1487691063 wait complete. 2024-04-25T18:52:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.911+0000 7f72a3172740 1 -- start start 2024-04-25T18:52:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 Processor -- start 2024-04-25T18:52:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559ac19d59e0 con 0x559ac19c7c30 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559ac18e3310 con 0x559ac19c8550 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559ac19d5b60 con 0x559ac19d2e10 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.912+0000 7f72a3172740 10 client.0 _ll_register_callbacks cb 0x559ac18b1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45890/0 (socket says 172.21.15.63:45890) 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 -- 172.21.15.63:0/1493680589 learned_addr learned my addr 172.21.15.63:0/1493680589 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729e200640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729ce00640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 -- 172.21.15.63:0/1493680589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 msgr2=0x559ac19d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 -- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 msgr2=0x559ac19d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 -- 172.21.15.63:0/1493680589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac1708620 con 0x559ac19c7c30 2024-04-25T18:52:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729ce00640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f729d800640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f728c09c7f0 tx=0x7f728c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ee0399b8a6eac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f727f400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.913+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c09c450 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72840054f0 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f728c0a52d0 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f728c0ce720 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 10 client.15729 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ac19e1c30 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72840054f0 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f728c0b8070 con 0x559ac19c7c30 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 10 client.15729 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 20 client.15729 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.914+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac19e30f0 0x559ac19e5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.915+0000 7f72a3172740 4 client.15729 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.915+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ac193d8b0 con 0x559ac19e30f0 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.915+0000 7f729ce00640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac19e30f0 0x559ac19e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.915+0000 7f729ce00640 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac19e30f0 0x559ac19e5540 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1d20845c0f5ae5c server_cookie=943a1ad214879a4a in_seq=0 out_seq=0 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.915+0000 7f727f400640 10 client.15729 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.916+0000 7f727f400640 1 -- 172.21.15.63:0/1493680589 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559ac193d8b0 con 0x559ac19e30f0 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.916+0000 7f727f400640 10 client.15729 handle_command_reply: tid=0 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.916+0000 7f72a3172740 1 client.15729 shutdown 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 msgr2=0x559ac19cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f728c09c7f0 tx=0x7f728c005740 comp rx=0 tx=0).stop 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 shutdown_connections 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac19e30f0 0x559ac19e5540 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 --2- 172.21.15.63:0/1493680589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19cfd40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 >> 172.21.15.63:0/1493680589 conn(0x559ac19c4140 msgr2=0x559ac19bee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f727f400640 0 client.15729 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 shutdown_connections 2024-04-25T18:52:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 1 -- 172.21.15.63:0/1493680589 wait complete. 2024-04-25T18:52:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.917+0000 7f72a3172740 20 client.15729 trim_cache size 0 max 16384 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 Processor -- start 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 -- start start 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19de8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19eae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72981ae1c0 con 0x559ac19c8550 2024-04-25T18:52:19.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72981aba30 con 0x559ac19c7c30 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.934+0000 7f72a3172740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72981a9880 con 0x559ac19d2e10 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45892/0 (socket says 172.21.15.63:45892) 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729ce00640 1 -- 172.21.15.63:0/277750164 learned_addr learned my addr 172.21.15.63:0/277750164 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729e200640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19de8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 -- 172.21.15.63:0/277750164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 msgr2=0x559ac19eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 -- 172.21.15.63:0/277750164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 msgr2=0x559ac19ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729d800640 1 -- 172.21.15.63:0/277750164 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ac19d5540 con 0x559ac19c7c30 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.935+0000 7f729ce00640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f729d800640 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19de8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f728c09ca80 tx=0x7f728c09b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b7927002ffb47f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f728a000640 1 -- 172.21.15.63:0/277750164 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c09c8c0 con 0x559ac19c7c30 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f728a000640 1 -- 172.21.15.63:0/277750164 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f728c09d1f0 con 0x559ac19c7c30 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f728a000640 1 -- 172.21.15.63:0/277750164 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c0ce520 con 0x559ac19c7c30 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f72a3172740 1 -- 172.21.15.63:0/277750164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 msgr2=0x559ac19de8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.936+0000 7f72a3172740 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19de8a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f728c09ca80 tx=0x7f728c09b000 comp rx=0 tx=0).stop 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 -- 172.21.15.63:0/277750164 shutdown_connections 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac19eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c8550 0x559ac19ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 --2- 172.21.15.63:0/277750164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c7c30 0x559ac19de8a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 -- 172.21.15.63:0/277750164 >> 172.21.15.63:0/277750164 conn(0x559ac19c4140 msgr2=0x559ac19bee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 -- 172.21.15.63:0/277750164 shutdown_connections 2024-04-25T18:52:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 -- 172.21.15.63:0/277750164 wait complete. 2024-04-25T18:52:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.937+0000 7f72a3172740 1 -- start start 2024-04-25T18:52:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 Processor -- start 2024-04-25T18:52:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac1a476c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72981a9ef0 con 0x559ac19c7c30 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f72981ab360 con 0x559ac19c8550 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f72981aa5c0 con 0x559ac19d2e10 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f72a3172740 10 client.0 _ll_register_callbacks cb 0x7f728c005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f729ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f729ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40188/0 (socket says 172.21.15.63:40188) 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f729d800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f729ce00640 1 -- 172.21.15.63:0/721713709 learned_addr learned my addr 172.21.15.63:0/721713709 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.938+0000 7f729e200640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac1a476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 -- 172.21.15.63:0/721713709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 msgr2=0x559ac1a476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac1a476c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 -- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 msgr2=0x559ac19d84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:19.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 -- 172.21.15.63:0/721713709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ac1a47c50 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729ce00640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f729d800640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19d7f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x559ac19cab30 tx=0x7f728c0a54e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec700955ad4240 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f727f400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f728c09c8c0 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f728c09b9b0 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.939+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f728c09d1f0 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f728c09caf0 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 10 client.15735 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559ac1a47c50 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f728c09d1f0 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f728c0b8050 con 0x559ac19c7c30 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 10 client.15735 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:19.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 20 client.15735 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac1a491a0 0x559ac1a4b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 4 client.15735 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.940+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ac1937320 con 0x559ac1a491a0 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.941+0000 7f729ce00640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac1a491a0 0x559ac1a4b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.941+0000 7f729ce00640 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac1a491a0 0x559ac1a4b5c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d415164b18843cb6 server_cookie=5de6f85355d7e98e in_seq=0 out_seq=0 2024-04-25T18:52:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.941+0000 7f727f400640 10 client.15735 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.962+0000 7f727f400640 1 -- 172.21.15.63:0/721713709 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ac1937320 con 0x559ac1a491a0 2024-04-25T18:52:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.962+0000 7f727f400640 10 client.15735 handle_command_reply: tid=0 2024-04-25T18:52:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.962+0000 7f72a3172740 1 client.15735 shutdown 2024-04-25T18:52:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 msgr2=0x559ac19d7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19d7f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x559ac19cab30 tx=0x7f728c0a54e0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 shutdown_connections 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559ac1a491a0 0x559ac1a4b5c0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559ac19c8550 0x559ac19d84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559ac19d2e10 0x559ac1a476c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 --2- 172.21.15.63:0/721713709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559ac19c7c30 0x559ac19d7f70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f727f400640 0 client.15735 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 >> 172.21.15.63:0/721713709 conn(0x559ac19c4140 msgr2=0x559ac19d0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 shutdown_connections 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 1 -- 172.21.15.63:0/721713709 wait complete. 2024-04-25T18:52:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.963+0000 7f72a3172740 20 client.15735 trim_cache size 0 max 16384 2024-04-25T18:52:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:52:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:52:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:52:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 msgr2=0x7f72800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 0x7f72800a6300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7290005fd0 tx=0x7f7290009500 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 msgr2=0x7f729819e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f728c00e9f0 tx=0x7f728c00edf0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 shutdown_connections 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f72800a3e40 0x7f72800a6300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7298103b40 0x7f729819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f729810c560 0x7f72981985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 --2- 172.21.15.63:0/71742055 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7298103220 0x7f729819e3f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 >> 172.21.15.63:0/71742055 conn(0x7f7298077fd0 msgr2=0x7f72980fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 shutdown_connections 2024-04-25T18:52:19.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:19.964+0000 7f72a0200640 1 -- 172.21.15.63:0/71742055 wait complete. 2024-04-25T18:52:20.392 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:20 smithi063 ceph-mon[28006]: pgmap v177: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-25T18:52:20.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.38 MB/sec execute 139 sec latency 25167.853 ms 2024-04-25T18:52:20.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:20 smithi114 ceph-mon[32988]: pgmap v177: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-25T18:52:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:20 smithi070 ceph-mon[32820]: pgmap v177: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 1.1 MiB/s wr, 53 op/s 2024-04-25T18:52:21.463 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.35 MB/sec execute 140 sec latency 26167.925 ms 2024-04-25T18:52:21.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 -- 172.21.15.63:0/2381965867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004074c70 msgr2=0x7fb004075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 --2- 172.21.15.63:0/2381965867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004074c70 0x7fb004075050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faff4009990 tx=0x7faff4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 -- 172.21.15.63:0/2381965867 shutdown_connections 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 --2- 172.21.15.63:0/2381965867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb004111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 --2- 172.21.15.63:0/2381965867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004074c70 0x7fb004075050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 --2- 172.21.15.63:0/2381965867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004075590 0x7fb00410ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 -- 172.21.15.63:0/2381965867 >> 172.21.15.63:0/2381965867 conn(0x7fb004079b10 msgr2=0x7fb004079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 -- 172.21.15.63:0/2381965867 shutdown_connections 2024-04-25T18:52:21.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.726+0000 7fb00b000700 1 -- 172.21.15.63:0/2381965867 wait complete. 2024-04-25T18:52:21.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.727+0000 7fb00b000700 1 Processor -- start 2024-04-25T18:52:21.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 -- start start 2024-04-25T18:52:21.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:21.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 0x7fb00419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:21.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb00419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:21.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb004114e70 con 0x7fb00410f470 2024-04-25T18:52:21.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb004114cf0 con 0x7fb004075590 2024-04-25T18:52:21.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.728+0000 7fb00b000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb004114ff0 con 0x7fb004074c70 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:53812/0 (socket says 172.21.15.63:53812) 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003e00700 1 -- 172.21.15.63:0/2368394034 learned_addr learned my addr 172.21.15.63:0/2368394034 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb009000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb00419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 0x7fb00419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 -- 172.21.15.63:0/2368394034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 msgr2=0x7fb00419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 -- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 msgr2=0x7fb00419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb00419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003400700 1 -- 172.21.15.63:0/2368394034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0041a0310 con 0x7fb004075590 2024-04-25T18:52:21.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb009000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb00419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.729+0000 7fb003e00700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb003400700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 0x7fb00419b840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faff000e9c0 tx=0x7faff000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0004220 con 0x7fb004075590 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff00043c0 con 0x7fb004075590 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0041a0600 con 0x7fb004075590 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff0003c50 con 0x7fb004075590 2024-04-25T18:52:21.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.730+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0041a7e80 con 0x7fb004075590 2024-04-25T18:52:21.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.732+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb004069730 con 0x7fb004075590 2024-04-25T18:52:21.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.734+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faff000c7c0 con 0x7fb004075590 2024-04-25T18:52:21.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.736+0000 7fb000c00700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 0x7fafe80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:21.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.736+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7faff00d0a00 con 0x7fb004075590 2024-04-25T18:52:21.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.736+0000 7fb003e00700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 0x7fafe80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:21.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.737+0000 7fb003e00700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 0x7fafe80a62f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faff4006010 tx=0x7faff401a040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:21.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:21.739+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faff009d030 con 0x7fb004075590 2024-04-25T18:52:22.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.027+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0041a80b0 con 0x7fb004075590 2024-04-25T18:52:22.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.028+0000 7fb000c00700 1 -- 172.21.15.63:0/2368394034 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11871 (secure 0 0 0) 0x7faff009c930 con 0x7fb004075590 2024-04-25T18:52:22.030 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:52:22.030 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:51:51.161805+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:52:22.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 msgr2=0x7fafe80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:22.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 0x7fafe80a62f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faff4006010 tx=0x7faff401a040 comp rx=0 tx=0).stop 2024-04-25T18:52:22.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 msgr2=0x7fb00419b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:22.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 0x7fb00419b840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faff000e9c0 tx=0x7faff000ecd0 comp rx=0 tx=0).stop 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 shutdown_connections 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fafe80a3e30 0x7fafe80a62f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb004074c70 0x7fb00419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00410f470 0x7fb00419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 --2- 172.21.15.63:0/2368394034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb004075590 0x7fb00419b840 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 >> 172.21.15.63:0/2368394034 conn(0x7fb004079b10 msgr2=0x7fb0040ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:22.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.033+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 shutdown_connections 2024-04-25T18:52:22.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:22.034+0000 7fb00b000700 1 -- 172.21.15.63:0/2368394034 wait complete. 2024-04-25T18:52:22.037 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 32 2024-04-25T18:52:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:22 smithi063 ceph-mon[28006]: pgmap v178: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:22.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:22 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2368394034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:52:22.464 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.32 MB/sec execute 141 sec latency 27168.329 ms 2024-04-25T18:52:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:22 smithi114 ceph-mon[32988]: pgmap v178: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:22 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2368394034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:52:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:22 smithi070 ceph-mon[32820]: pgmap v178: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:22 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2368394034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:52:22.929 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id bd1a99b1 --release --await' 2024-04-25T18:52:22.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id bd1a99b1 --release --await 2024-04-25T18:52:23.395 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:52:23.464 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.29 MB/sec execute 142 sec latency 28168.460 ms 2024-04-25T18:52:24.464 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.26 MB/sec execute 143 sec latency 29168.565 ms 2024-04-25T18:52:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:24 smithi114 ceph-mon[32988]: pgmap v179: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:24.616 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:24 smithi063 ceph-mon[28006]: pgmap v179: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:24 smithi070 ceph-mon[32820]: pgmap v179: 97 pgs: 97 active+clean; 55 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1.0 MiB/s wr, 42 op/s 2024-04-25T18:52:25.464 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177505 4.23 MB/sec execute 144 sec latency 30168.651 ms 2024-04-25T18:52:25.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 -- 172.21.15.63:0/2337367711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 msgr2=0x7f702406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 --2- 172.21.15.63:0/2337367711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f702406afb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f70140098f0 tx=0x7f7014009c00 comp rx=0 tx=0).stop 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 -- 172.21.15.63:0/2337367711 shutdown_connections 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 --2- 172.21.15.63:0/2337367711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70241053c0 0x7f7024105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 --2- 172.21.15.63:0/2337367711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f702406afb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 --2- 172.21.15.63:0/2337367711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f702406b4f0 0x7f702406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 -- 172.21.15.63:0/2337367711 >> 172.21.15.63:0/2337367711 conn(0x7f7024078780 msgr2=0x7f7024078b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 -- 172.21.15.63:0/2337367711 shutdown_connections 2024-04-25T18:52:25.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.835+0000 7f7023e00700 1 -- 172.21.15.63:0/2337367711 wait complete. 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.836+0000 7f7023e00700 1 Processor -- start 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.836+0000 7f7023e00700 1 -- start start 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 0x7f702419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 0x7f702419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f70241a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70241190c0 con 0x7f7024107920 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7024118f40 con 0x7f702406b4f0 2024-04-25T18:52:25.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7024119240 con 0x7f70241053c0 2024-04-25T18:52:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f70241a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7022000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 0x7f702419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f70241a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45946/0 (socket says 172.21.15.63:45946) 2024-04-25T18:52:25.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7022a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 0x7f702419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7022000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 0x7f702419fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:53826/0 (socket says 172.21.15.63:53826) 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.837+0000 7f7023400700 1 -- 172.21.15.63:0/1023286435 learned_addr learned my addr 172.21.15.63:0/1023286435 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 -- 172.21.15.63:0/1023286435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 msgr2=0x7f702419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 0x7f702419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 -- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 msgr2=0x7f70241a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f70241a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 -- 172.21.15.63:0/1023286435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70241a4590 con 0x7f702406b4f0 2024-04-25T18:52:25.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7022a00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 0x7f702419f5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f701400c010 tx=0x7f701400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.838+0000 7f7028a00700 1 -- 172.21.15.63:0/1023286435 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140050c0 con 0x7f702406b4f0 2024-04-25T18:52:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.839+0000 7f7028a00700 1 -- 172.21.15.63:0/1023286435 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701400b560 con 0x7f702406b4f0 2024-04-25T18:52:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.839+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70241a4820 con 0x7f702406b4f0 2024-04-25T18:52:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.839+0000 7f7028a00700 1 -- 172.21.15.63:0/1023286435 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7014003a00 con 0x7f702406b4f0 2024-04-25T18:52:25.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.839+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241ac100 con 0x7f702406b4f0 2024-04-25T18:52:25.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.842+0000 7f7028a00700 1 -- 172.21.15.63:0/1023286435 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f701400b710 con 0x7f702406b4f0 2024-04-25T18:52:25.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.843+0000 7f7028a00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 0x7f70080a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 Processor -- start 2024-04-25T18:52:25.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7028a00700 1 -- 172.21.15.63:0/1023286435 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f70140d1550 con 0x7f702406b4f0 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7022000700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 0x7f70080a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 -- start start 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede45e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45e92c0 0x55ede45eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ede4530f90 con 0x55ede45de410 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ede44d7830 con 0x55ede45e92c0 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.844+0000 7f7034814b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ede452cf50 con 0x55ede45ddaf0 2024-04-25T18:52:25.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:53842/0 (socket says 172.21.15.63:53842) 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 -- 172.21.15.63:0/1463472086 learned_addr learned my addr 172.21.15.63:0/1463472086 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022000700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede45e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7023400700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45e92c0 0x55ede45eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022000700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 0x7f70080a62f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7024101be0 tx=0x7f7010009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 -- 172.21.15.63:0/1463472086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45e92c0 msgr2=0x55ede45eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45e92c0 0x55ede45eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 -- 172.21.15.63:0/1463472086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 msgr2=0x55ede45e8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede45e8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022a00700 1 -- 172.21.15.63:0/1463472086 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ede44cd780 con 0x55ede45ddaf0 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.845+0000 7f7022000700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede45e8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.846+0000 7f7022a00700 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7014099ee0 tx=0x7f70140998e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb9b6b7142ebf788 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.846+0000 7f7018c00700 1 -- 172.21.15.63:0/1463472086 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701409dc10 con 0x55ede45ddaf0 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.846+0000 7f7018c00700 1 -- 172.21.15.63:0/1463472086 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701409d510 con 0x55ede45ddaf0 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.846+0000 7f7018c00700 1 -- 172.21.15.63:0/1463472086 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701409d770 con 0x55ede45ddaf0 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 -- 172.21.15.63:0/1463472086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 msgr2=0x55ede45dded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7014099ee0 tx=0x7f70140998e0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 -- 172.21.15.63:0/1463472086 shutdown_connections 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45dded0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45e92c0 0x55ede45eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 --2- 172.21.15.63:0/1463472086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede45e8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 -- 172.21.15.63:0/1463472086 >> 172.21.15.63:0/1463472086 conn(0x55ede450c6a0 msgr2=0x55ede45d9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 -- 172.21.15.63:0/1463472086 shutdown_connections 2024-04-25T18:52:25.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.847+0000 7f7034814b80 1 -- 172.21.15.63:0/1463472086 wait complete. 2024-04-25T18:52:25.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.848+0000 7f7034814b80 1 -- start start 2024-04-25T18:52:25.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.848+0000 7f7034814b80 1 Processor -- start 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.848+0000 7f7034814b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 0x55ede45e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ede45ebbf0 con 0x55ede45e92c0 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ede44ea960 con 0x55ede45de410 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ede45ebd70 con 0x55ede45ddaf0 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7034814b80 10 client.0 _ll_register_callbacks cb 0x55ede44c9820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7023400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 0x55ede45e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7023400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 0x55ede45e62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45956/0 (socket says 172.21.15.63:45956) 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7023400700 1 -- 172.21.15.63:0/1715090847 learned_addr learned my addr 172.21.15.63:0/1715090847 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022000700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 -- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 msgr2=0x55ede45e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45e5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 -- 172.21.15.63:0/1715090847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 msgr2=0x55ede45e62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 0x55ede45e62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.849+0000 7f7022a00700 1 -- 172.21.15.63:0/1715090847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ede447e9d0 con 0x55ede45ddaf0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f7022a00700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45e8a60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55ede45dae30 tx=0x7f701409ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd61449e0446fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f700d600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140ba070 con 0x55ede45ddaf0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f701409d770 con 0x55ede45ddaf0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f701409d510 con 0x55ede45ddaf0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f70140a18c0 con 0x55ede45ddaf0 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f7034814b80 10 client.25951 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:25.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.850+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ede44c7bf0 con 0x55ede45ddaf0 2024-04-25T18:52:25.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701409d510 con 0x55ede45ddaf0 2024-04-25T18:52:25.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f70140a5cf0 con 0x55ede45ddaf0 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 10 client.25951 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 10 client.25951 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 20 client.25951 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede45f8740 0x55ede45fab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 4 client.25951 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.851+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ede453e780 con 0x55ede45f8740 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.852+0000 7f7022000700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede45f8740 0x55ede45fab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.852+0000 7f7022000700 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede45f8740 0x55ede45fab30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a715d79079b42ef server_cookie=bb33e6ddac272363 in_seq=0 out_seq=0 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.852+0000 7f700d600700 10 client.25951 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f700d600700 1 -- 172.21.15.63:0/1715090847 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ede453e780 con 0x55ede45f8740 2024-04-25T18:52:25.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f700d600700 10 client.25951 handle_command_reply: tid=0 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 client.25951 shutdown 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 msgr2=0x55ede45e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45e8a60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55ede45dae30 tx=0x7f701409ba20 comp rx=0 tx=0).stop 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 shutdown_connections 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede45f8740 0x55ede45fab30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45ddaf0 0x55ede45e8a60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45e92c0 0x55ede45e62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f7034814b80 1 --2- 172.21.15.63:0/1715090847 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.853+0000 7f700d600700 0 client.25951 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.854+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 >> 172.21.15.63:0/1715090847 conn(0x55ede450c6a0 msgr2=0x55ede45dfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.854+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 shutdown_connections 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.854+0000 7f7034814b80 1 -- 172.21.15.63:0/1715090847 wait complete. 2024-04-25T18:52:25.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.854+0000 7f7034814b80 20 client.25951 trim_cache size 0 max 16384 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 Processor -- start 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 -- start start 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45ddaf0 0x55ede45d3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede4600340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b6080 con 0x55ede45de410 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b38b0 con 0x55ede45ddaf0 2024-04-25T18:52:25.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7034814b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b1700 con 0x55ede45e92c0 2024-04-25T18:52:25.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7022000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede4600340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.877+0000 7f7023400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:53854/0 (socket says 172.21.15.63:53854) 2024-04-25T18:52:25.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 -- 172.21.15.63:0/3237450802 learned_addr learned my addr 172.21.15.63:0/3237450802 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7022a00700 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45ddaf0 0x55ede45d3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 -- 172.21.15.63:0/3237450802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45ddaf0 msgr2=0x55ede45d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45ddaf0 0x55ede45d3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 -- 172.21.15.63:0/3237450802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 msgr2=0x55ede4600340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede4600340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 -- 172.21.15.63:0/3237450802 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ede45e42d0 con 0x55ede45e92c0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.878+0000 7f7023400700 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f701c0075c0 tx=0x7f701c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=835ac2f56a01ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f700fe00700 1 -- 172.21.15.63:0/3237450802 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c01b070 con 0x55ede45e92c0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f700fe00700 1 -- 172.21.15.63:0/3237450802 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701c020910 con 0x55ede45e92c0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f700fe00700 1 -- 172.21.15.63:0/3237450802 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c017a40 con 0x55ede45e92c0 2024-04-25T18:52:25.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 -- 172.21.15.63:0/3237450802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 msgr2=0x55ede4600880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f701c0075c0 tx=0x7f701c00b040 comp rx=0 tx=0).stop 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 -- 172.21.15.63:0/3237450802 shutdown_connections 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede4600880 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45ddaf0 0x55ede45d3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 --2- 172.21.15.63:0/3237450802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45de410 0x55ede4600340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.879+0000 7f7034814b80 1 -- 172.21.15.63:0/3237450802 >> 172.21.15.63:0/3237450802 conn(0x55ede450c6a0 msgr2=0x55ede45e61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.880+0000 7f7034814b80 1 -- 172.21.15.63:0/3237450802 shutdown_connections 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.880+0000 7f7034814b80 1 -- 172.21.15.63:0/3237450802 wait complete. 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.880+0000 7f7034814b80 1 -- start start 2024-04-25T18:52:25.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.880+0000 7f7034814b80 1 Processor -- start 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 0x55ede45f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede45efd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b1d70 con 0x55ede45ddaf0 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b31e0 con 0x55ede45de410 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f70241b2440 con 0x55ede45e92c0 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7034814b80 10 client.0 _ll_register_callbacks cb 0x55ede44c9820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7022a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 0x55ede45f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7022000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7022000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40268/0 (socket says 172.21.15.63:40268) 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7022000700 1 -- 172.21.15.63:0/2199281337 learned_addr learned my addr 172.21.15.63:0/2199281337 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.881+0000 7f7023400700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede45efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 -- 172.21.15.63:0/2199281337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 msgr2=0x55ede45efd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede45efd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 -- 172.21.15.63:0/2199281337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 msgr2=0x55ede45f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 0x55ede45f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 -- 172.21.15.63:0/2199281337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ede45f05c0 con 0x55ede45de410 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022a00700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 0x55ede45f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7022000700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7010011fd0 tx=0x7f70100047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d01cc7cee29414cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f700d600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701002d070 con 0x55ede45de410 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f701c020910 con 0x55ede45de410 2024-04-25T18:52:25.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.882+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7f70100298e0 con 0x55ede45de410 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7010038050 con 0x55ede45de410 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 10 client.25940 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ede465fdb0 con 0x55ede45de410 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701c020910 con 0x55ede45de410 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7f7010005020 con 0x55ede45de410 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 10 client.25940 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 10 client.25940 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 20 client.25940 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede4660500 0x55ede4662920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 4 client.25940 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "bd1a99b1", "release": true, "await": true}] 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.883+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "bd1a99b1", "release": true, "await": true}) -- 0x55ede4538820 con 0x55ede4660500 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.884+0000 7f7022a00700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede4660500 0x55ede4662920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.884+0000 7f7022a00700 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede4660500 0x55ede4662920 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ef2fa1de83a5a21 server_cookie=c76515a75d50144a in_seq=0 out_seq=0 2024-04-25T18:52:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.884+0000 7f700d600700 10 client.25940 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f700d600700 1 -- 172.21.15.63:0/2199281337 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55ede4538820 con 0x55ede4660500 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f700d600700 10 client.25940 handle_command_reply: tid=0 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 client.25940 shutdown 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 msgr2=0x55ede45ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7010011fd0 tx=0x7f70100047c0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 shutdown_connections 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ede4660500 0x55ede4662920 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ede45e92c0 0x55ede45efd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ede45ddaf0 0x55ede45f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 --2- 172.21.15.63:0/2199281337 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ede45de410 0x55ede45ef7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 >> 172.21.15.63:0/2199281337 conn(0x55ede450c6a0 msgr2=0x55ede45e7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.888+0000 7f700d600700 0 client.25940 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.889+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 shutdown_connections 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.889+0000 7f7034814b80 1 -- 172.21.15.63:0/2199281337 wait complete. 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.889+0000 7f7034814b80 20 client.25940 trim_cache size 0 max 16384 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:52:25.891 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 32, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 20, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "bd1a99b1": { 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 20, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 82.3, 2024-04-25T18:52:25.892 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:52:25.893 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 msgr2=0x7f70080a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 0x7f70080a62f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7024101be0 tx=0x7f7010009260 comp rx=0 tx=0).stop 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 msgr2=0x7f702419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 0x7f702419f5d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f701400c010 tx=0x7f701400fa00 comp rx=0 tx=0).stop 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 shutdown_connections 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f70080a3e30 0x7f70080a62f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7024107920 0x7f70241a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f702406b4f0 0x7f702419f5d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 --2- 172.21.15.63:0/1023286435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f70241053c0 0x7f702419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:25.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 >> 172.21.15.63:0/1023286435 conn(0x7f7024078780 msgr2=0x7f7024100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:25.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.890+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 shutdown_connections 2024-04-25T18:52:25.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:52:25.891+0000 7f7023e00700 1 -- 172.21.15.63:0/1023286435 wait complete. 2024-04-25T18:52:26.464 INFO:tasks.workunit.client.0.smithi063.stdout: 1 177948 4.21 MB/sec execute 145 sec latency 30598.583 ms 2024-04-25T18:52:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:26 smithi114 ceph-mon[32988]: pgmap v180: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-04-25T18:52:26.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:26 smithi063 ceph-mon[28006]: pgmap v180: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-04-25T18:52:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:26 smithi070 ceph-mon[32820]: pgmap v180: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 1.1 MiB/s wr, 51 op/s 2024-04-25T18:52:26.935 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'bd1a99b1', seconds elapsed: 39.2 2024-04-25T18:52:27.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 179061 4.21 MB/sec execute 146 sec latency 18.555 ms 2024-04-25T18:52:28.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 180633 4.25 MB/sec execute 147 sec latency 17.320 ms 2024-04-25T18:52:28.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:28 smithi114 ceph-mon[32988]: pgmap v181: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 59 KiB/s wr, 19 op/s 2024-04-25T18:52:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:28 smithi063 ceph-mon[28006]: pgmap v181: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 59 KiB/s wr, 19 op/s 2024-04-25T18:52:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:28 smithi070 ceph-mon[32820]: pgmap v181: 97 pgs: 97 active+clean; 55 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 59 KiB/s wr, 19 op/s 2024-04-25T18:52:29.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 181527 4.25 MB/sec execute 148 sec latency 16.941 ms 2024-04-25T18:52:30.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 182891 4.26 MB/sec execute 149 sec latency 24.762 ms 2024-04-25T18:52:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:30 smithi114 ceph-mon[32988]: pgmap v182: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 335 KiB/s wr, 61 op/s 2024-04-25T18:52:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:30 smithi063 ceph-mon[28006]: pgmap v182: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 335 KiB/s wr, 61 op/s 2024-04-25T18:52:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:30 smithi070 ceph-mon[32820]: pgmap v182: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 335 KiB/s wr, 61 op/s 2024-04-25T18:52:31.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 184454 4.31 MB/sec execute 150 sec latency 15.051 ms 2024-04-25T18:52:32.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185323 4.30 MB/sec execute 151 sec latency 19.488 ms 2024-04-25T18:52:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:32 smithi114 ceph-mon[32988]: pgmap v183: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:32 smithi063 ceph-mon[28006]: pgmap v183: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:32 smithi070 ceph-mon[32820]: pgmap v183: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 -- 172.21.15.63:0/819108408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 msgr2=0x7fef14110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 --2- 172.21.15.63:0/819108408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef14110ea0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009be0 tx=0x7fef0c02a6c0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 -- 172.21.15.63:0/819108408 shutdown_connections 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 --2- 172.21.15.63:0/819108408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104bb0 0x7fef14108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 --2- 172.21.15.63:0/819108408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef14110ea0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 --2- 172.21.15.63:0/819108408 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef14104200 0x7fef141045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 -- 172.21.15.63:0/819108408 >> 172.21.15.63:0/819108408 conn(0x7fef140ffa60 msgr2=0x7fef14101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.112+0000 7fef13200640 1 -- 172.21.15.63:0/819108408 shutdown_connections 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.113+0000 7fef13200640 1 -- 172.21.15.63:0/819108408 wait complete. 2024-04-25T18:52:33.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 Processor -- start 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 -- start start 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 0x7fef1419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef1419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef14114220 con 0x7fef141092d0 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef141140a0 con 0x7fef14104200 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.114+0000 7fef13200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef141143a0 con 0x7fef1419f020 2024-04-25T18:52:33.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57690/0 (socket says 172.21.15.63:57690) 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 -- 172.21.15.63:0/489287386 learned_addr learned my addr 172.21.15.63:0/489287386 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef11e00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 0x7fef1419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef0be00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef1419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 -- 172.21.15.63:0/489287386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 msgr2=0x7fef1419a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 0x7fef1419a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 -- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 msgr2=0x7fef1419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef1419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 -- 172.21.15.63:0/489287386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1419fb10 con 0x7fef1419f020 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef11e00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 0x7fef1419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef0be00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef1419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:52:33.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.115+0000 7fef12800640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009b30 tx=0x7fef0c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.116+0000 7fef0aa00640 1 -- 172.21.15.63:0/489287386 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c040020 con 0x7fef1419f020 2024-04-25T18:52:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.116+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1419fda0 con 0x7fef1419f020 2024-04-25T18:52:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.116+0000 7fef0aa00640 1 -- 172.21.15.63:0/489287386 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef0c030970 con 0x7fef1419f020 2024-04-25T18:52:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.116+0000 7fef0aa00640 1 -- 172.21.15.63:0/489287386 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c036040 con 0x7fef1419f020 2024-04-25T18:52:33.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.116+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef141a75c0 con 0x7fef1419f020 2024-04-25T18:52:33.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.119+0000 7fef0aa00640 1 -- 172.21.15.63:0/489287386 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fef0c043020 con 0x7fef1419f020 2024-04-25T18:52:33.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.119+0000 7fef1ba25740 1 Processor -- start 2024-04-25T18:52:33.122 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 -- start start 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f89650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f8bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d219ee4900 con 0x55d219f89b90 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d219e8aef0 con 0x55d219f7e9b0 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef1ba25740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d219ee08c0 con 0x55d219f7f2d0 2024-04-25T18:52:33.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef0be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef0be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38472/0 (socket says 172.21.15.63:38472) 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef0be00640 1 -- 172.21.15.63:0/1927758766 learned_addr learned my addr 172.21.15.63:0/1927758766 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef11e00640 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f89650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.120+0000 7fef0be00640 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f8bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 -- 172.21.15.63:0/1927758766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 msgr2=0x55d219f89650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f89650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 -- 172.21.15.63:0/1927758766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 msgr2=0x55d219f8bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f8bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 -- 172.21.15.63:0/1927758766 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d219bf7210 con 0x55d219f7e9b0 2024-04-25T18:52:33.125 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0aa00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 0x7feef80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0be00640 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feefc0093e0 tx=0x7feefc0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342af8ab91ac5ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef0aa00640 1 -- 172.21.15.63:0/489287386 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fef0c0f1560 con 0x7fef1419f020 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef03000640 1 -- 172.21.15.63:0/1927758766 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc012cc0 con 0x55d219f7e9b0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef11e00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 0x7feef80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef03000640 1 -- 172.21.15.63:0/1927758766 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feefc0107d0 con 0x55d219f7e9b0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.121+0000 7fef03000640 1 -- 172.21.15.63:0/1927758766 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc012e60 con 0x55d219f7e9b0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 -- 172.21.15.63:0/1927758766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 msgr2=0x55d219f7ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feefc0093e0 tx=0x7feefc0096c0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef11e00640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 0x7feef80a61d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55d219f7ff50 tx=0x7fef04009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 -- 172.21.15.63:0/1927758766 shutdown_connections 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f8bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f89650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.126 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1927758766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f7ed90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 -- 172.21.15.63:0/1927758766 >> 172.21.15.63:0/1927758766 conn(0x55d219f7aec0 msgr2=0x55d219f7b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 -- 172.21.15.63:0/1927758766 shutdown_connections 2024-04-25T18:52:33.127 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.122+0000 7fef1ba25740 1 -- 172.21.15.63:0/1927758766 wait complete. 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.123+0000 7fef1ba25740 1 -- start start 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.123+0000 7fef1ba25740 1 Processor -- start 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f87090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d219f8c7c0 con 0x55d219f89b90 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d219e9e310 con 0x55d219f7e9b0 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d219f8c940 con 0x55d219f7f2d0 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef1ba25740 10 client.0 _ll_register_callbacks cb 0x55d219e6ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38476/0 (socket says 172.21.15.63:38476) 2024-04-25T18:52:33.130 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 -- 172.21.15.63:0/4121569063 learned_addr learned my addr 172.21.15.63:0/4121569063 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef12800640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef0be00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f87090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 -- 172.21.15.63:0/4121569063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 msgr2=0x55d219f87090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f87090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 -- 172.21.15.63:0/4121569063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 msgr2=0x55d219f875d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f875d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef11e00640 1 -- 172.21.15.63:0/4121569063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d219cc3620 con 0x55d219f7e9b0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.124+0000 7fef12800640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f875d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7fef11e00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fef040045e0 tx=0x7fef04004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2043cb2e04c8ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7feef7400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04003dc0 con 0x55d219f7e9b0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fef0402a420 con 0x55d219f7e9b0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feefc0107d0 con 0x55d219f7e9b0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fef0402a620 con 0x55d219f7e9b0 2024-04-25T18:52:33.131 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7fef1ba25740 10 client.25952 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.125+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d219f98af0 con 0x55d219f7e9b0 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feefc0107d0 con 0x55d219f7e9b0 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fef04021940 con 0x55d219f7e9b0 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 10 client.25952 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 10 client.25952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 20 client.25952 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d219f9a070 0x55d219f9c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 4 client.25952 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d219ef88b0 con 0x55d219f9a070 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.126+0000 7fef0be00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d219f9a070 0x55d219f9c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.127+0000 7fef0be00640 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d219f9a070 0x55d219f9c4f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1983ae21c26a69 server_cookie=43dcded2a3df6c8e in_seq=0 out_seq=0 2024-04-25T18:52:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.127+0000 7feef7400640 10 client.25952 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7feef7400640 1 -- 172.21.15.63:0/4121569063 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d219ef88b0 con 0x55d219f9a070 2024-04-25T18:52:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7feef7400640 10 client.25952 handle_command_reply: tid=0 2024-04-25T18:52:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 client.25952 shutdown 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 msgr2=0x55d219f86b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fef040045e0 tx=0x7fef04004730 comp rx=0 tx=0).stop 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 shutdown_connections 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d219f9a070 0x55d219f9c4f0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219f875d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7f2d0 0x55d219f87090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 --2- 172.21.15.63:0/4121569063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7e9b0 0x55d219f86b50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7feef7400640 0 client.25952 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 >> 172.21.15.63:0/4121569063 conn(0x55d219f7aec0 msgr2=0x55d219f80bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 shutdown_connections 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.128+0000 7fef1ba25740 1 -- 172.21.15.63:0/4121569063 wait complete. 2024-04-25T18:52:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.129+0000 7fef1ba25740 20 client.25952 trim_cache size 0 max 16384 2024-04-25T18:52:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.145+0000 7fef1ba25740 1 Processor -- start 2024-04-25T18:52:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.145+0000 7fef1ba25740 1 -- start start 2024-04-25T18:52:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.145+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.150 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 0x55d219f81370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219fa1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef1ba25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef141b2550 con 0x55d219f89b90 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef1ba25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef141afdc0 con 0x55d219f7f2d0 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef1ba25740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef141adc10 con 0x55d219f7e9b0 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57716/0 (socket says 172.21.15.63:57716) 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 -- 172.21.15.63:0/1663816466 learned_addr learned my addr 172.21.15.63:0/1663816466 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef12800640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219fa1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef0be00640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 0x55d219f81370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 -- 172.21.15.63:0/1663816466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 msgr2=0x55d219f81370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 0x55d219f81370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 -- 172.21.15.63:0/1663816466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 msgr2=0x55d219fa1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219fa1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef11e00640 1 -- 172.21.15.63:0/1663816466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d219f87ab0 con 0x55d219f7e9b0 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.146+0000 7fef0be00640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 0x55d219f81370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.147+0000 7fef11e00640 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fef04012d20 tx=0x7fef0402a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9504c2d88340884 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.147+0000 7fef02600640 1 -- 172.21.15.63:0/1663816466 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04036070 con 0x55d219f7e9b0 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.147+0000 7fef02600640 1 -- 172.21.15.63:0/1663816466 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04025030 con 0x55d219f7e9b0 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.147+0000 7fef02600640 1 -- 172.21.15.63:0/1663816466 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04021410 con 0x55d219f7e9b0 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- 172.21.15.63:0/1663816466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 msgr2=0x55d219f95680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fef04012d20 tx=0x7fef0402a5b0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- 172.21.15.63:0/1663816466 shutdown_connections 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f7f2d0 0x55d219f81370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f89b90 0x55d219fa1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 --2- 172.21.15.63:0/1663816466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219f95680 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- 172.21.15.63:0/1663816466 >> 172.21.15.63:0/1663816466 conn(0x55d219f7aec0 msgr2=0x55d219f873a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- 172.21.15.63:0/1663816466 shutdown_connections 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- 172.21.15.63:0/1663816466 wait complete. 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 -- start start 2024-04-25T18:52:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.148+0000 7fef1ba25740 1 Processor -- start 2024-04-25T18:52:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219ff89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 0x55d219ff8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f89b90 0x55d219f8ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef141ae280 con 0x55d219f7f2d0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef141af6f0 con 0x55d219f89b90 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fef141ae950 con 0x55d219f7e9b0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef1ba25740 10 client.0 _ll_register_callbacks cb 0x7fef040044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef0be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 0x55d219ff8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef11e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219ff89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef0be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 0x55d219ff8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42760/0 (socket says 172.21.15.63:42760) 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.149+0000 7fef0be00640 1 -- 172.21.15.63:0/3336487382 learned_addr learned my addr 172.21.15.63:0/3336487382 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef12800640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f89b90 0x55d219f8ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 -- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f89b90 msgr2=0x55d219f8ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f89b90 0x55d219f8ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 -- 172.21.15.63:0/3336487382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 msgr2=0x55d219ff8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 0x55d219ff8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 -- 172.21.15.63:0/3336487382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d219f8f620 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef11e00640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219ff89b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fef04021d00 tx=0x7fef0402ad90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c91c0acb7bb4c80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7feef7400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04048060 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fef04005030 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.150+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef04025030 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fef04036070 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 10 client.25975 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d21a0000d0 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04025030 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7fef04021860 con 0x55d219f7e9b0 2024-04-25T18:52:33.157 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 10 client.25975 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 10 client.25975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 20 client.25975 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d21a0006e0 0x55d21a002b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 4 client.25975 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.151+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d219ef2320 con 0x55d21a0006e0 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.152+0000 7fef0be00640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d21a0006e0 0x55d21a002b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.152+0000 7fef0be00640 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d21a0006e0 0x55d21a002b00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a331f115bad9e25 server_cookie=191e30a9b5a16a8f in_seq=0 out_seq=0 2024-04-25T18:52:33.158 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.152+0000 7feef7400640 10 client.25975 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7feef7400640 1 -- 172.21.15.63:0/3336487382 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d219ef2320 con 0x55d21a0006e0 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7feef7400640 10 client.25975 handle_command_reply: tid=0 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 client.25975 shutdown 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 msgr2=0x55d219ff89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219ff89b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fef04021d00 tx=0x7fef0402ad90 comp rx=0 tx=0).stop 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 shutdown_connections 2024-04-25T18:52:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d21a0006e0 0x55d21a002b00 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d219f89b90 0x55d219f8ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d219f7f2d0 0x55d219ff8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 --2- 172.21.15.63:0/3336487382 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d219f7e9b0 0x55d219ff89b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7feef7400640 0 client.25975 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.192+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 >> 172.21.15.63:0/3336487382 conn(0x55d219f7aec0 msgr2=0x55d219f79790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.193+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 shutdown_connections 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.193+0000 7fef1ba25740 1 -- 172.21.15.63:0/3336487382 wait complete. 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.193+0000 7fef1ba25740 20 client.25975 trim_cache size 0 max 16384 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:52:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:52:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:52:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:52:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 msgr2=0x7feef80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 0x7feef80a61d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x55d219f7ff50 tx=0x7fef04009260 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 msgr2=0x7fef1419f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009b30 tx=0x7fef0c003c10 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 shutdown_connections 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feef80a3d10 0x7feef80a61d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fef1419f020 0x7fef1419f4d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef141092d0 0x7fef1419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 --2- 172.21.15.63:0/489287386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef14104200 0x7fef1419a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 >> 172.21.15.63:0/489287386 conn(0x7fef140ffa60 msgr2=0x7fef14101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 shutdown_connections 2024-04-25T18:52:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:33.194+0000 7fef13200640 1 -- 172.21.15.63:0/489287386 wait complete. 2024-04-25T18:52:33.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.30 MB/sec execute 152 sec latency 652.013 ms 2024-04-25T18:52:34.465 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.27 MB/sec execute 153 sec latency 1652.173 ms 2024-04-25T18:52:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:34 smithi114 ceph-mon[32988]: pgmap v184: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:34 smithi063 ceph-mon[28006]: pgmap v184: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:34 smithi070 ceph-mon[32820]: pgmap v184: 97 pgs: 97 active+clean; 58 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 298 KiB/s wr, 51 op/s 2024-04-25T18:52:35.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.24 MB/sec execute 154 sec latency 2652.339 ms 2024-04-25T18:52:36.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.22 MB/sec execute 155 sec latency 3652.497 ms 2024-04-25T18:52:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:36 smithi114 ceph-mon[32988]: pgmap v185: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 126 op/s 2024-04-25T18:52:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:36 smithi063 ceph-mon[28006]: pgmap v185: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 126 op/s 2024-04-25T18:52:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:36 smithi070 ceph-mon[32820]: pgmap v185: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 708 KiB/s wr, 126 op/s 2024-04-25T18:52:37.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.19 MB/sec execute 156 sec latency 4652.703 ms 2024-04-25T18:52:38.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.16 MB/sec execute 157 sec latency 5652.857 ms 2024-04-25T18:52:38.571 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:38 smithi114 ceph-mon[32988]: pgmap v186: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 687 KiB/s wr, 117 op/s 2024-04-25T18:52:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:38 smithi063 ceph-mon[28006]: pgmap v186: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 687 KiB/s wr, 117 op/s 2024-04-25T18:52:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:38 smithi070 ceph-mon[32820]: pgmap v186: 97 pgs: 97 active+clean; 58 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 687 KiB/s wr, 117 op/s 2024-04-25T18:52:39.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.14 MB/sec execute 158 sec latency 6652.954 ms 2024-04-25T18:52:40.466 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.11 MB/sec execute 159 sec latency 7653.114 ms 2024-04-25T18:52:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:40 smithi114 ceph-mon[32988]: pgmap v187: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 702 KiB/s wr, 130 op/s 2024-04-25T18:52:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:40 smithi063 ceph-mon[28006]: pgmap v187: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 702 KiB/s wr, 130 op/s 2024-04-25T18:52:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:40 smithi070 ceph-mon[32820]: pgmap v187: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 702 KiB/s wr, 130 op/s 2024-04-25T18:52:41.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.08 MB/sec execute 160 sec latency 8653.277 ms 2024-04-25T18:52:42.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.06 MB/sec execute 161 sec latency 9653.430 ms 2024-04-25T18:52:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:42 smithi063 ceph-mon[28006]: pgmap v188: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:42 smithi070 ceph-mon[32820]: pgmap v188: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:42 smithi114 ceph-mon[32988]: pgmap v188: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:43.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.03 MB/sec execute 162 sec latency 10653.574 ms 2024-04-25T18:52:44.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 4.01 MB/sec execute 163 sec latency 11653.708 ms 2024-04-25T18:52:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:44 smithi063 ceph-mon[28006]: pgmap v189: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:44 smithi070 ceph-mon[32820]: pgmap v189: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:44 smithi114 ceph-mon[32988]: pgmap v189: 97 pgs: 97 active+clean; 58 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 425 KiB/s wr, 88 op/s 2024-04-25T18:52:45.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 185857 3.98 MB/sec execute 164 sec latency 12653.851 ms 2024-04-25T18:52:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 -- 172.21.15.63:0/3999606421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0102b50 msgr2=0x7fedf0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 --2- 172.21.15.63:0/3999606421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0102b50 0x7fedf0106b90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feddc009900 tx=0x7feddc02a3f0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 -- 172.21.15.63:0/3999606421 shutdown_connections 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 --2- 172.21.15.63:0/3999606421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0102b50 0x7fedf0106b90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 --2- 172.21.15.63:0/3999606421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf0107270 0x7fedf010ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 --2- 172.21.15.63:0/3999606421 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 -- 172.21.15.63:0/3999606421 >> 172.21.15.63:0/3999606421 conn(0x7fedf00793e0 msgr2=0x7fedf00797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.350+0000 7fedf5c00640 1 -- 172.21.15.63:0/3999606421 shutdown_connections 2024-04-25T18:52:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.351+0000 7fedf5c00640 1 -- 172.21.15.63:0/3999606421 wait complete. 2024-04-25T18:52:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 Processor -- start 2024-04-25T18:52:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 -- start start 2024-04-25T18:52:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0107270 0x7fedf01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 0x7fedf019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0112200 con 0x7fedf019a720 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0112080 con 0x7fedf0107270 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedf5c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fedf0112380 con 0x7fedf01021a0 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.352+0000 7fedef400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 0x7fedf019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0107270 0x7fedf01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedef400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 0x7fedf019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37534/0 (socket says 172.21.15.63:37534) 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedef400640 1 -- 172.21.15.63:0/2322224761 learned_addr learned my addr 172.21.15.63:0/2322224761 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 -- 172.21.15.63:0/2322224761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0107270 msgr2=0x7fedf01a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0107270 0x7fedf01a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 -- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 msgr2=0x7fedf019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 0x7fedf019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 -- 172.21.15.63:0/2322224761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf019b320 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.353+0000 7fedeea00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf01a0660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fede000e9f0 tx=0x7fede000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.354+0000 7fede7400640 1 -- 172.21.15.63:0/2322224761 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede000ccc0 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.354+0000 7fede7400640 1 -- 172.21.15.63:0/2322224761 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fede0005020 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.354+0000 7fede7400640 1 -- 172.21.15.63:0/2322224761 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede0004a00 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.354+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf01a7400 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.354+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf01a77b0 con 0x7fedf01021a0 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.357+0000 7fedf8a32740 1 Processor -- start 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.357+0000 7fedf8a32740 1 -- start start 2024-04-25T18:52:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.357+0000 7fede7400640 1 -- 172.21.15.63:0/2322224761 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fede0004ba0 con 0x7fedf01021a0 2024-04-25T18:52:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b2dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557972a93900 con 0x557972b2d820 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557972a39ef0 con 0x557972b38a00 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedf8a32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557972a8f8c0 con 0x557972b2e140 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedef400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b2dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38624/0 (socket says 172.21.15.63:38624) 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.358+0000 7fedee000640 1 -- 172.21.15.63:0/730020092 learned_addr learned my addr 172.21.15.63:0/730020092 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 -- 172.21.15.63:0/730020092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 msgr2=0x557972b384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 -- 172.21.15.63:0/730020092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 msgr2=0x557972b2dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b2dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 -- 172.21.15.63:0/730020092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579727a6210 con 0x557972b38a00 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fedee000640 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feddc006010 tx=0x7feddc004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc683b48cc7a4c8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7feddbe00640 1 -- 172.21.15.63:0/730020092 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc035040 con 0x557972b38a00 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7fede7400640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 0x7fedd40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7feddbe00640 1 -- 172.21.15.63:0/730020092 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc004430 con 0x557972b38a00 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.359+0000 7feddbe00640 1 -- 172.21.15.63:0/730020092 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc0035f0 con 0x557972b38a00 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fede7400640 1 -- 172.21.15.63:0/2322224761 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7fede00d0740 con 0x7fedf01021a0 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedef400640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 0x7fedd40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 -- 172.21.15.63:0/730020092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 msgr2=0x557972b3adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feddc006010 tx=0x7feddc004040 comp rx=0 tx=0).stop 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 -- 172.21.15.63:0/730020092 shutdown_connections 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b3adf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 --2- 172.21.15.63:0/730020092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b2dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.360+0000 7fedf8a32740 1 -- 172.21.15.63:0/730020092 >> 172.21.15.63:0/730020092 conn(0x557972b29d30 msgr2=0x557972b2a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.361+0000 7fedf8a32740 1 -- 172.21.15.63:0/730020092 shutdown_connections 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.361+0000 7fedef400640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 0x7fedd40a6300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fede80049d0 tx=0x7fede8009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.361+0000 7fedf8a32740 1 -- 172.21.15.63:0/730020092 wait complete. 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.361+0000 7fedf8a32740 1 -- start start 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.361+0000 7fedf8a32740 1 Processor -- start 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b36460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557972b3b5d0 con 0x557972b2d820 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557972a4d310 con 0x557972b38a00 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557972b3b750 con 0x557972b2e140 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedf8a32740 10 client.0 _ll_register_callbacks cb 0x557972a1bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b36460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedef400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b36460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38632/0 (socket says 172.21.15.63:38632) 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56298/0 (socket says 172.21.15.63:56298) 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedef400640 1 -- 172.21.15.63:0/2723985474 learned_addr learned my addr 172.21.15.63:0/2723985474 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedeea00640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 -- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 msgr2=0x557972b36460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b36460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 -- 172.21.15.63:0/2723985474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 msgr2=0x557972b359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.362+0000 7fedee000640 1 -- 172.21.15.63:0/2723985474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557972872620 con 0x557972b2e140 2024-04-25T18:52:46.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedeea00640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:52:46.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedee000640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x557972b2b150 tx=0x7feddc004980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=348b46da758752c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc053060 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7feddc003800 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.363+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feddc004430 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feddc035070 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 10 client.25987 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557972b3b5d0 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc004430 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7feddc031b40 con 0x557972b2e140 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 10 client.25987 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 20 client.25987 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972b48a00 0x557972b4adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 4 client.25987 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557972aa78b0 con 0x557972b48a00 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.364+0000 7fedeea00640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972b48a00 0x557972b4adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.365+0000 7fedeea00640 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972b48a00 0x557972b4adf0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a3eeda949cecc12 server_cookie=6025fe183c0365ce in_seq=0 out_seq=0 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.365+0000 7fedd8c00640 10 client.25987 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedd8c00640 1 -- 172.21.15.63:0/2723985474 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557972aa78b0 con 0x557972b48a00 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedd8c00640 10 client.25987 handle_command_reply: tid=0 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 client.25987 shutdown 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 msgr2=0x557972b35f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x557972b2b150 tx=0x7feddc004980 comp rx=0 tx=0).stop 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 shutdown_connections 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972b48a00 0x557972b4adf0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b38a00 0x557972b36460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2e140 0x557972b35f20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2723985474 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b2d820 0x557972b359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedd8c00640 0 client.25987 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 >> 172.21.15.63:0/2723985474 conn(0x557972b29d30 msgr2=0x557972b2fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 shutdown_connections 2024-04-25T18:52:46.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 1 -- 172.21.15.63:0/2723985474 wait complete. 2024-04-25T18:52:46.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.366+0000 7fedf8a32740 20 client.25987 trim_cache size 0 max 16384 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.384+0000 7fedf8a32740 1 Processor -- start 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.384+0000 7fedf8a32740 1 -- start start 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b360b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b50510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01b2600 con 0x557972b38a00 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01afe70 con 0x557972b2e140 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedf8a32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01adcc0 con 0x557972b2d820 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56308/0 (socket says 172.21.15.63:56308) 2024-04-25T18:52:46.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedeea00640 1 -- 172.21.15.63:0/590809226 learned_addr learned my addr 172.21.15.63:0/590809226 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedef400640 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b50510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 -- 172.21.15.63:0/590809226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 msgr2=0x557972b44490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 -- 172.21.15.63:0/590809226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 msgr2=0x557972b50510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b50510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedee000640 1 -- 172.21.15.63:0/590809226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557972b33f30 con 0x557972b2e140 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.385+0000 7fedeea00640 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.386+0000 7fedee000640 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b360b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feddc02a9e0 tx=0x7feddc005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdbeb665f07ac1e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.386+0000 7feddb400640 1 -- 172.21.15.63:0/590809226 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc053070 con 0x557972b2e140 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.386+0000 7feddb400640 1 -- 172.21.15.63:0/590809226 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc04a070 con 0x557972b2e140 2024-04-25T18:52:46.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.386+0000 7feddb400640 1 -- 172.21.15.63:0/590809226 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc003c30 con 0x557972b2e140 2024-04-25T18:52:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- 172.21.15.63:0/590809226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 msgr2=0x557972b360b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b360b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feddc02a9e0 tx=0x7feddc005a20 comp rx=0 tx=0).stop 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- 172.21.15.63:0/590809226 shutdown_connections 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b360b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b50510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 --2- 172.21.15.63:0/590809226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b44490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- 172.21.15.63:0/590809226 >> 172.21.15.63:0/590809226 conn(0x557972b29d30 msgr2=0x557972b359a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- 172.21.15.63:0/590809226 shutdown_connections 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- 172.21.15.63:0/590809226 wait complete. 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 -- start start 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.387+0000 7fedf8a32740 1 Processor -- start 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b40680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b3ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01ae330 con 0x557972b38a00 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01af7a0 con 0x557972b2e140 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fedf01aea00 con 0x557972b2d820 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedf8a32740 10 client.0 _ll_register_callbacks cb 0x7feddc005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedee000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38644/0 (socket says 172.21.15.63:38644) 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedeea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b40680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedee000640 1 -- 172.21.15.63:0/2268424695 learned_addr learned my addr 172.21.15.63:0/2268424695 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.388+0000 7fedef400640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b3ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 -- 172.21.15.63:0/2268424695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 msgr2=0x557972b40680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b40680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 -- 172.21.15.63:0/2268424695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 msgr2=0x557972b3ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b3ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 -- 172.21.15.63:0/2268424695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557972b3f2a0 con 0x557972b2e140 2024-04-25T18:52:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedeea00640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b40680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedee000640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feddc005ab0 tx=0x7feddc004460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46a5d82749c8ba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedd8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc053040 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7feddc04eb60 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feddc04a070 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.389+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feddc06f050 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 10 client.25973 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557972baf830 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc04a070 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7feddc0314a0 con 0x557972b2e140 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 10 client.25973 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 10 client.25973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 20 client.25973 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972bafef0 0x557972bb2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 4 client.25973 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:52:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557972aa1320 con 0x557972bafef0 2024-04-25T18:52:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.390+0000 7fedeea00640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972bafef0 0x557972bb2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.391+0000 7fedeea00640 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972bafef0 0x557972bb2310 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=324c8e52b6f653e3 server_cookie=9c3c0251de8b0d81 in_seq=0 out_seq=0 2024-04-25T18:52:46.396 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.391+0000 7fedd8c00640 10 client.25973 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedd8c00640 1 -- 172.21.15.63:0/2268424695 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557972aa1320 con 0x557972bafef0 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedd8c00640 10 client.25973 handle_command_reply: tid=0 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 client.25973 shutdown 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 msgr2=0x557972b40bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feddc005ab0 tx=0x7feddc004460 comp rx=0 tx=0).stop 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 shutdown_connections 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557972bafef0 0x557972bb2310 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557972b2e140 0x557972b40bc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557972b38a00 0x557972b3ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.408+0000 7fedf8a32740 1 --2- 172.21.15.63:0/2268424695 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557972b2d820 0x557972b40680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.409+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 >> 172.21.15.63:0/2268424695 conn(0x557972b29d30 msgr2=0x557972b36210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.409+0000 7fedd8c00640 0 client.25973 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.409+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 shutdown_connections 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.409+0000 7fedf8a32740 1 -- 172.21.15.63:0/2268424695 wait complete. 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.409+0000 7fedf8a32740 20 client.25973 trim_cache size 0 max 16384 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:52:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 msgr2=0x7fedd40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 0x7fedd40a6300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fede80049d0 tx=0x7fede8009500 comp rx=0 tx=0).stop 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 msgr2=0x7fedf01a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf01a0660 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fede000e9f0 tx=0x7fede000eeb0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 shutdown_connections 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fedd40a3e40 0x7fedd40a6300 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fedf0107270 0x7fedf01a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fedf019a720 0x7fedf019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 --2- 172.21.15.63:0/2322224761 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fedf01021a0 0x7fedf01a0660 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 >> 172.21.15.63:0/2322224761 conn(0x7fedf00793e0 msgr2=0x7fedf00ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 shutdown_connections 2024-04-25T18:52:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:46.410+0000 7fedf5c00640 1 -- 172.21.15.63:0/2322224761 wait complete. 2024-04-25T18:52:46.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 186105 3.96 MB/sec execute 165 sec latency 13402.934 ms 2024-04-25T18:52:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:46 smithi063 ceph-mon[28006]: pgmap v190: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 516 KiB/s wr, 95 op/s 2024-04-25T18:52:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:46 smithi070 ceph-mon[32820]: pgmap v190: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 516 KiB/s wr, 95 op/s 2024-04-25T18:52:46.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:46 smithi114 ceph-mon[32988]: pgmap v190: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 516 KiB/s wr, 95 op/s 2024-04-25T18:52:47.467 INFO:tasks.workunit.client.0.smithi063.stdout: 1 187453 3.98 MB/sec execute 166 sec latency 21.077 ms 2024-04-25T18:52:48.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 188542 4.00 MB/sec execute 167 sec latency 15.693 ms 2024-04-25T18:52:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:48 smithi063 ceph-mon[28006]: pgmap v191: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 106 KiB/s wr, 20 op/s 2024-04-25T18:52:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:48 smithi070 ceph-mon[32820]: pgmap v191: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 106 KiB/s wr, 20 op/s 2024-04-25T18:52:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:48 smithi114 ceph-mon[32988]: pgmap v191: 97 pgs: 97 active+clean; 59 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 106 KiB/s wr, 20 op/s 2024-04-25T18:52:49.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 189512 4.00 MB/sec execute 168 sec latency 22.504 ms 2024-04-25T18:52:50.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 190785 4.01 MB/sec execute 169 sec latency 21.925 ms 2024-04-25T18:52:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:50 smithi063 ceph-mon[28006]: pgmap v192: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 371 KiB/s wr, 59 op/s 2024-04-25T18:52:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:50 smithi070 ceph-mon[32820]: pgmap v192: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 371 KiB/s wr, 59 op/s 2024-04-25T18:52:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:50 smithi114 ceph-mon[32988]: pgmap v192: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 371 KiB/s wr, 59 op/s 2024-04-25T18:52:51.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 192050 4.04 MB/sec execute 170 sec latency 18.000 ms 2024-04-25T18:52:51.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:51 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:51.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:51 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:51.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:51 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:52:52.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 193239 4.05 MB/sec execute 171 sec latency 25.549 ms 2024-04-25T18:52:52.605 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:52 smithi070 ceph-mon[32820]: pgmap v193: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:52.605 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:52 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:52.605 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:52 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:52.605 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:52 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:52:52.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:52 smithi063 ceph-mon[28006]: pgmap v193: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:52.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:52 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:52.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:52 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:52.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:52 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:52 smithi114 ceph-mon[32988]: pgmap v193: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:52 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:52 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:52:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:52 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:52:53.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 194742 4.07 MB/sec execute 172 sec latency 21.743 ms 2024-04-25T18:52:54.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 195664 4.08 MB/sec execute 173 sec latency 17.400 ms 2024-04-25T18:52:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:54 smithi063 ceph-mon[28006]: pgmap v194: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:54 smithi070 ceph-mon[32820]: pgmap v194: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:54 smithi114 ceph-mon[32988]: pgmap v194: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 355 KiB/s wr, 45 op/s 2024-04-25T18:52:55.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 196602 4.08 MB/sec execute 174 sec latency 22.699 ms 2024-04-25T18:52:56.468 INFO:tasks.workunit.client.0.smithi063.stdout: 1 197813 4.10 MB/sec execute 175 sec latency 22.506 ms 2024-04-25T18:52:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:56 smithi063 ceph-mon[28006]: pgmap v195: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 KiB/s wr, 103 op/s 2024-04-25T18:52:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:56 smithi070 ceph-mon[32820]: pgmap v195: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 KiB/s wr, 103 op/s 2024-04-25T18:52:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:56 smithi114 ceph-mon[32988]: pgmap v195: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 KiB/s wr, 103 op/s 2024-04-25T18:52:57.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 198893 4.12 MB/sec execute 176 sec latency 37.273 ms 2024-04-25T18:52:58.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 199822 4.11 MB/sec execute 177 sec latency 21.616 ms 2024-04-25T18:52:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:52:58 smithi063 ceph-mon[28006]: pgmap v196: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 KiB/s wr, 96 op/s 2024-04-25T18:52:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:52:58 smithi070 ceph-mon[32820]: pgmap v196: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 KiB/s wr, 96 op/s 2024-04-25T18:52:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:52:58 smithi114 ceph-mon[32988]: pgmap v196: 97 pgs: 97 active+clean; 66 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 689 KiB/s wr, 96 op/s 2024-04-25T18:52:59.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 201082 4.11 MB/sec execute 178 sec latency 26.360 ms 2024-04-25T18:52:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 -- 172.21.15.63:0/433725359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe4101fe0 msgr2=0x7febe41023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 --2- 172.21.15.63:0/433725359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe4101fe0 0x7febe41023c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7febd4009900 tx=0x7febd402a410 comp rx=0 tx=0).stop 2024-04-25T18:52:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 -- 172.21.15.63:0/433725359 shutdown_connections 2024-04-25T18:52:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 --2- 172.21.15.63:0/433725359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe41070b0 0x7febe410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 --2- 172.21.15.63:0/433725359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe4102990 0x7febe41069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 --2- 172.21.15.63:0/433725359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe4101fe0 0x7febe41023c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 -- 172.21.15.63:0/433725359 >> 172.21.15.63:0/433725359 conn(0x7febe40fd8a0 msgr2=0x7febe40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 -- 172.21.15.63:0/433725359 shutdown_connections 2024-04-25T18:52:59.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.564+0000 7febe9a00640 1 -- 172.21.15.63:0/433725359 wait complete. 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 Processor -- start 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 -- start start 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe41070b0 0x7febe41a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 0x7febe419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7febe4112010 con 0x7febe41070b0 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febe4111e90 con 0x7febe4101fe0 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.565+0000 7febe9a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7febe4112190 con 0x7febe419a630 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.568 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57568/0 (socket says 172.21.15.63:57568) 2024-04-25T18:52:59.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 -- 172.21.15.63:0/2357334732 learned_addr learned my addr 172.21.15.63:0/2357334732 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:59.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe9000640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 0x7febe419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3400640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe41070b0 0x7febe41a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 -- 172.21.15.63:0/2357334732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 msgr2=0x7febe419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 0x7febe419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 -- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe41070b0 msgr2=0x7febe41a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe41070b0 0x7febe41a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe3e00640 1 -- 172.21.15.63:0/2357334732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe419b230 con 0x7febe4101fe0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.566+0000 7febe9000640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 0x7febe419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe3e00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7febd4005940 tx=0x7febd40040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe0c00640 1 -- 172.21.15.63:0/2357334732 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd403f030 con 0x7febe4101fe0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe419b4c0 con 0x7febe4101fe0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe0c00640 1 -- 172.21.15.63:0/2357334732 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febd40044c0 con 0x7febe4101fe0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe0c00640 1 -- 172.21.15.63:0/2357334732 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd4035070 con 0x7febe4101fe0 2024-04-25T18:52:59.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.567+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe41a7600 con 0x7febe4101fe0 2024-04-25T18:52:59.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.570+0000 7febe0c00640 1 -- 172.21.15.63:0/2357334732 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7febd4042030 con 0x7febe4101fe0 2024-04-25T18:52:59.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 Processor -- start 2024-04-25T18:52:59.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 -- start start 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316862650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x556316864f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5563167bd900 con 0x5563168582d0 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556316763ef0 con 0x556316862b90 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.571+0000 7febedfb5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5563167b98c0 con 0x5563168579b0 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51996/0 (socket says 172.21.15.63:51996) 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316862650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 -- 172.21.15.63:0/1176823360 learned_addr learned my addr 172.21.15.63:0/1176823360 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:59.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe3400640 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x556316864f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 -- 172.21.15.63:0/1176823360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 msgr2=0x556316864f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x556316864f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 -- 172.21.15.63:0/1176823360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 msgr2=0x556316862650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316862650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 -- 172.21.15.63:0/1176823360 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563164d0210 con 0x5563168579b0 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe3400640 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x556316864f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe0c00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 0x7febc80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.572+0000 7febe9000640 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7febdc011790 tx=0x7febdc011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c176f37f4affb6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.573+0000 7febe0c00640 1 -- 172.21.15.63:0/2357334732 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7febd40f0b80 con 0x7febe4101fe0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.573+0000 7febe3e00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 0x7febc80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.573+0000 7febd9600640 1 -- 172.21.15.63:0/1176823360 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc004d20 con 0x5563168579b0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.573+0000 7febd9600640 1 -- 172.21.15.63:0/1176823360 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febdc003970 con 0x5563168579b0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.573+0000 7febd9600640 1 -- 172.21.15.63:0/1176823360 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc004490 con 0x5563168579b0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- 172.21.15.63:0/1176823360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 msgr2=0x556316857d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7febdc011790 tx=0x7febdc011c50 comp rx=0 tx=0).stop 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febe3e00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 0x7febc80a61d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x556316858f50 tx=0x7febd40f4630 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- 172.21.15.63:0/1176823360 shutdown_connections 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x556316864f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316862650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 --2- 172.21.15.63:0/1176823360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316857d90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- 172.21.15.63:0/1176823360 >> 172.21.15.63:0/1176823360 conn(0x556316853ec0 msgr2=0x5563168542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- 172.21.15.63:0/1176823360 shutdown_connections 2024-04-25T18:52:59.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- 172.21.15.63:0/1176823360 wait complete. 2024-04-25T18:52:59.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.574+0000 7febedfb5740 1 -- start start 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 Processor -- start 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x5563168600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556316862b90 0x5563168605f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5563168657c0 con 0x5563168582d0 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556316777310 con 0x5563168579b0 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x556316865940 con 0x556316862b90 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febedfb5740 10 client.0 _ll_register_callbacks cb 0x556316745cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febe9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febe3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x5563168600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.575+0000 7febe9000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57574/0 (socket says 172.21.15.63:57574) 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe9000640 1 -- 172.21.15.63:0/189448781 learned_addr learned my addr 172.21.15.63:0/189448781 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:59.581 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe9000640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556316862b90 0x5563168605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 -- 172.21.15.63:0/189448781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556316862b90 msgr2=0x5563168605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556316862b90 0x5563168605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 -- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 msgr2=0x55631685fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 -- 172.21.15.63:0/189448781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55631659c620 con 0x5563168582d0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe9000640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febe3400640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x5563168600b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x556316855050 tx=0x7febcc012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c13e6dd8d11b5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.576+0000 7febd2000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc004030 con 0x5563168582d0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febdc003970 con 0x5563168582d0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7febcc0041d0 con 0x5563168582d0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febcc0043d0 con 0x5563168582d0 2024-04-25T18:52:59.582 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febedfb5740 10 client.15807 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556316871b20 con 0x5563168582d0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.577+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febdc003970 con 0x5563168582d0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7febcc003b90 con 0x5563168582d0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 10 client.15807 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 10 client.15807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 20 client.15807 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x556316872fe0 0x556316875410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 4 client.15807 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563167d18b0 con 0x556316872fe0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.578+0000 7febe9000640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x556316872fe0 0x556316875410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.579+0000 7febe9000640 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x556316872fe0 0x556316875410 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfb526d1dd3e24a9 server_cookie=719b1efc37b3a59e in_seq=0 out_seq=0 2024-04-25T18:52:59.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.579+0000 7febd2000640 10 client.15807 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:59.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febd2000640 1 -- 172.21.15.63:0/189448781 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563167d18b0 con 0x556316872fe0 2024-04-25T18:52:59.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febd2000640 10 client.15807 handle_command_reply: tid=0 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 client.15807 shutdown 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 msgr2=0x5563168600b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x5563168600b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x556316855050 tx=0x7febcc012b60 comp rx=0 tx=0).stop 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 shutdown_connections 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x556316872fe0 0x556316875410 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556316862b90 0x5563168605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x5563168600b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 --2- 172.21.15.63:0/189448781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168579b0 0x55631685fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febd2000640 0 client.15807 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 >> 172.21.15.63:0/189448781 conn(0x556316853ec0 msgr2=0x556316859c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 shutdown_connections 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.581+0000 7febedfb5740 1 -- 172.21.15.63:0/189448781 wait complete. 2024-04-25T18:52:59.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.582+0000 7febedfb5740 20 client.15807 trim_cache size 0 max 16384 2024-04-25T18:52:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.599+0000 7febedfb5740 1 Processor -- start 2024-04-25T18:52:59.601 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.599+0000 7febedfb5740 1 -- start start 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x55631686e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 0x55631687b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7febe41b25c0 con 0x556316862b90 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febe41afe30 con 0x5563168582d0 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febedfb5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7febe41adc80 con 0x5563168579b0 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57588/0 (socket says 172.21.15.63:57588) 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 -- 172.21.15.63:0/3641596790 learned_addr learned my addr 172.21.15.63:0/3641596790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe9000640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x55631686e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe9000640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 0x55631687b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 -- 172.21.15.63:0/3641596790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 msgr2=0x55631686e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x55631686e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 -- 172.21.15.63:0/3641596790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 msgr2=0x55631687b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 0x55631687b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe3400640 1 -- 172.21.15.63:0/3641596790 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55631685e0c0 con 0x5563168582d0 2024-04-25T18:52:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.600+0000 7febe9000640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 0x55631687b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:52:59.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.601+0000 7febe3400640 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7febcc012420 tx=0x7febcc009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142dc30540068671 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.601+0000 7febd8c00640 1 -- 172.21.15.63:0/3641596790 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc028030 con 0x5563168582d0 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.601+0000 7febd8c00640 1 -- 172.21.15.63:0/3641596790 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febcc00fda0 con 0x5563168582d0 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.601+0000 7febd8c00640 1 -- 172.21.15.63:0/3641596790 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc003b10 con 0x5563168582d0 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 -- 172.21.15.63:0/3641596790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 msgr2=0x55631687ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7febcc012420 tx=0x7febcc009640 comp rx=0 tx=0).stop 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 -- 172.21.15.63:0/3641596790 shutdown_connections 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5563168582d0 0x55631687ace0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556316862b90 0x55631687b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 --2- 172.21.15.63:0/3641596790 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x55631686e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 -- 172.21.15.63:0/3641596790 >> 172.21.15.63:0/3641596790 conn(0x556316853ec0 msgr2=0x55631685ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.605 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.602+0000 7febedfb5740 1 -- 172.21.15.63:0/3641596790 shutdown_connections 2024-04-25T18:52:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.603+0000 7febedfb5740 1 -- 172.21.15.63:0/3641596790 wait complete. 2024-04-25T18:52:59.606 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.603+0000 7febedfb5740 1 -- start start 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.603+0000 7febedfb5740 1 Processor -- start 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316867d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x5563168d7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7febe41ae2f0 con 0x5563168582d0 2024-04-25T18:52:59.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febe41af760 con 0x556316862b90 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7febe41ae9c0 con 0x5563168579b0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febedfb5740 10 client.0 _ll_register_callbacks cb 0x7febcc009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37412/0 (socket says 172.21.15.63:37412) 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 -- 172.21.15.63:0/1075224053 learned_addr learned my addr 172.21.15.63:0/1075224053 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe9000640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x5563168d7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe9000640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316867d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 -- 172.21.15.63:0/1075224053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 msgr2=0x556316867d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316867d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 -- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 msgr2=0x5563168d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x5563168d7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.604+0000 7febe3400640 1 -- 172.21.15.63:0/1075224053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563168d7ac0 con 0x5563168582d0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febe3400640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7febcc003fd0 tx=0x7febcc00f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa628ff4502b8d25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febd2000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:52:59.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc031020 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8543+0+0 (secure 0 0 0) 0x7febcc003910 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febcc00fda0 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febcc028070 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febedfb5740 10 client.15810 fetch_fsmap learned FSMap version 32 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.605+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5563168d8d40 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febcc00fda0 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 32) ==== 8346+0+0 (secure 0 0 0) 0x7febcc01c420 con 0x5563168582d0 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 10 client.15810 fetch_fsmap finished waiting for FSMap version 32 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 20 client.15810 populate_metadata read hostname 'smithi063' 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5563168d9470 0x5563168db890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 4 client.15810 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563167cb320 con 0x5563168d9470 2024-04-25T18:52:59.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.606+0000 7febe9000640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5563168d9470 0x5563168db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:52:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.607+0000 7febe9000640 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5563168d9470 0x5563168db890 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be965dda0980c159 server_cookie=d92ab36ea02d1a56 in_seq=0 out_seq=0 2024-04-25T18:52:59.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.607+0000 7febd2000640 10 client.15810 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.650+0000 7febd2000640 1 -- 172.21.15.63:0/1075224053 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563167cb320 con 0x5563168d9470 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.650+0000 7febd2000640 10 client.15810 handle_command_reply: tid=0 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.650+0000 7febedfb5740 1 client.15810 shutdown 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.650+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 msgr2=0x556316868290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.650+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7febcc003fd0 tx=0x7febcc00f970 comp rx=0 tx=0).stop 2024-04-25T18:52:59.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 shutdown_connections 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5563168d9470 0x5563168db890 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556316862b90 0x5563168d7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5563168582d0 0x556316868290 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febd2000640 0 client.15810 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 --2- 172.21.15.63:0/1075224053 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5563168579b0 0x556316867d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 >> 172.21.15.63:0/1075224053 conn(0x556316853ec0 msgr2=0x55631685fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 shutdown_connections 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 1 -- 172.21.15.63:0/1075224053 wait complete. 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.651+0000 7febedfb5740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T18:52:59.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 msgr2=0x7febc80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 0x7febc80a61d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x556316858f50 tx=0x7febd40f4630 comp rx=0 tx=0).stop 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 msgr2=0x7febe41a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:52:59.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7febd4005940 tx=0x7febd40040c0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 shutdown_connections 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7febc80a3d10 0x7febc80a61d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7febe419a630 0x7febe419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febe41070b0 0x7febe41a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 --2- 172.21.15.63:0/2357334732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febe4101fe0 0x7febe41a0570 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 >> 172.21.15.63:0/2357334732 conn(0x7febe40fd8a0 msgr2=0x7febe40fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 shutdown_connections 2024-04-25T18:52:59.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:52:59.652+0000 7febe9a00640 1 -- 172.21.15.63:0/2357334732 wait complete. 2024-04-25T18:53:00.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 202350 4.15 MB/sec execute 179 sec latency 33.044 ms 2024-04-25T18:53:00.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:00 smithi063 ceph-mon[28006]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-25T18:53:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:00 smithi070 ceph-mon[32820]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-25T18:53:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:00 smithi114 ceph-mon[32988]: pgmap v197: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-04-25T18:53:01.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 202734 4.13 MB/sec execute 180 sec latency 59.889 ms 2024-04-25T18:53:02.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 203705 4.14 MB/sec execute 181 sec latency 54.162 ms 2024-04-25T18:53:02.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:02 smithi063 ceph-mon[28006]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:02 smithi070 ceph-mon[32820]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:02.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:02 smithi114 ceph-mon[32988]: pgmap v198: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:03.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 205169 4.16 MB/sec execute 182 sec latency 22.381 ms 2024-04-25T18:53:04.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 206287 4.17 MB/sec execute 183 sec latency 18.982 ms 2024-04-25T18:53:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:04 smithi070 ceph-mon[32820]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:04 smithi114 ceph-mon[32988]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:04 smithi063 ceph-mon[28006]: pgmap v199: 97 pgs: 97 active+clean; 69 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 958 KiB/s wr, 114 op/s 2024-04-25T18:53:05.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 207302 4.17 MB/sec execute 184 sec latency 23.812 ms 2024-04-25T18:53:05.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:05 smithi070 ceph-mon[32820]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T18:53:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:05 smithi114 ceph-mon[32988]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T18:53:05.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:05 smithi063 ceph-mon[28006]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T18:53:06.469 INFO:tasks.workunit.client.0.smithi063.stdout: 1 208659 4.19 MB/sec execute 185 sec latency 21.267 ms 2024-04-25T18:53:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:06 smithi070 ceph-mon[32820]: pgmap v201: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-04-25T18:53:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:06 smithi114 ceph-mon[32988]: pgmap v201: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-04-25T18:53:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:06 smithi063 ceph-mon[28006]: pgmap v201: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 236 op/s 2024-04-25T18:53:07.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 209830 4.21 MB/sec execute 186 sec latency 18.442 ms 2024-04-25T18:53:08.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 210935 4.21 MB/sec execute 187 sec latency 29.403 ms 2024-04-25T18:53:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:08 smithi070 ceph-mon[32820]: pgmap v202: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 237 op/s 2024-04-25T18:53:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:08 smithi114 ceph-mon[32988]: pgmap v202: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 237 op/s 2024-04-25T18:53:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:08 smithi063 ceph-mon[28006]: pgmap v202: 97 pgs: 97 active+clean; 69 MiB data, 967 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 237 op/s 2024-04-25T18:53:09.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 212250 4.23 MB/sec execute 188 sec latency 25.823 ms 2024-04-25T18:53:10.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 213237 4.24 MB/sec execute 189 sec latency 18.198 ms 2024-04-25T18:53:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:10 smithi070 ceph-mon[32820]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:10.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:10 smithi114 ceph-mon[32988]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:10 smithi063 ceph-mon[28006]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:11.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214220 4.24 MB/sec execute 190 sec latency 23.374 ms 2024-04-25T18:53:12.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.22 MB/sec execute 191 sec latency 293.990 ms 2024-04-25T18:53:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:12 smithi070 ceph-mon[32820]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:12 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 -- 172.21.15.63:0/2995750242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac8108e10 msgr2=0x7f0ac8115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 --2- 172.21.15.63:0/2995750242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac8108e10 0x7f0ac8115250 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009cd0 tx=0x7f0ac402a8f0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 -- 172.21.15.63:0/2995750242 shutdown_connections 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 --2- 172.21.15.63:0/2995750242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 --2- 172.21.15.63:0/2995750242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac8108e10 0x7f0ac8115250 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 --2- 172.21.15.63:0/2995750242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac810e420 0x7f0ac810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.798+0000 7f0acf200640 1 -- 172.21.15.63:0/2995750242 >> 172.21.15.63:0/2995750242 conn(0x7f0ac8077d30 msgr2=0x7f0ac8106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.799+0000 7f0acf200640 1 -- 172.21.15.63:0/2995750242 shutdown_connections 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.799+0000 7f0acf200640 1 -- 172.21.15.63:0/2995750242 wait complete. 2024-04-25T18:53:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 Processor -- start 2024-04-25T18:53:12.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 -- start start 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac8108e10 0x7f0ac81a4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac810e420 0x7f0ac819e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac8118650 con 0x7f0ac8108e10 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81184d0 con 0x7f0ac8108490 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acf200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81187d0 con 0x7f0ac810e420 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acd400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac8108e10 0x7f0ac81a4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acde00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55846/0 (socket says 172.21.15.63:55846) 2024-04-25T18:53:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.800+0000 7f0acde00640 1 -- 172.21.15.63:0/3865121552 learned_addr learned my addr 172.21.15.63:0/3865121552 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0ace800640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac810e420 0x7f0ac819e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 -- 172.21.15.63:0/3865121552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac810e420 msgr2=0x7f0ac819e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac810e420 0x7f0ac819e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 -- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac8108e10 msgr2=0x7f0ac81a4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac8108e10 0x7f0ac81a4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 -- 172.21.15.63:0/3865121552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac819f0d0 con 0x7f0ac8108490 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.801+0000 7f0acde00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00e9f0 tx=0x7f0abc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.802+0000 7f0ac2a00640 1 -- 172.21.15.63:0/3865121552 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc00cd80 con 0x7f0ac8108490 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.802+0000 7f0ac2a00640 1 -- 172.21.15.63:0/3865121552 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0abc0108e0 con 0x7f0ac8108490 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.802+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac819f3c0 con 0x7f0ac8108490 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.802+0000 7f0ac2a00640 1 -- 172.21.15.63:0/3865121552 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc018980 con 0x7f0ac8108490 2024-04-25T18:53:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.802+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac81ab660 con 0x7f0ac8108490 2024-04-25T18:53:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.806+0000 7f0ad376c740 1 Processor -- start 2024-04-25T18:53:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.806+0000 7f0ad376c740 1 -- start start 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e8f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e900650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e34e85b900 con 0x55e34e8f62d0 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e34e801ef0 con 0x55e34e900b90 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ad376c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e34e8578c0 con 0x55e34e8f59b0 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ac2a00640 1 -- 172.21.15.63:0/3865121552 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0abc010430 con 0x7f0ac8108490 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0acd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0acd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55860/0 (socket says 172.21.15.63:55860) 2024-04-25T18:53:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0acd400640 1 -- 172.21.15.63:0/75302387 learned_addr learned my addr 172.21.15.63:0/75302387 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ace800640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e8f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ace800640 1 -- 172.21.15.63:0/75302387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 msgr2=0x55e34e902f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ace800640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0ace800640 1 -- 172.21.15.63:0/75302387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 msgr2=0x55e34e900650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.807+0000 7f0acde00640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e900650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ace800640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e900650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ace800640 1 -- 172.21.15.63:0/75302387 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e34e56e210 con 0x55e34e8f59b0 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0acd400640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0acde00640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e900650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:53:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ace800640 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e8f5d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0ac402ae10 tx=0x7f0ac40093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ea45250b1baef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ab7000640 1 -- 172.21.15.63:0/75302387 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4030620 con 0x55e34e8f59b0 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ab7000640 1 -- 172.21.15.63:0/75302387 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac4036030 con 0x55e34e8f59b0 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.808+0000 7f0ab7000640 1 -- 172.21.15.63:0/75302387 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4004330 con 0x55e34e8f59b0 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ac2a00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 0x7f0ab00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ac2a00640 1 -- 172.21.15.63:0/3865121552 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8631+0+0 (secure 0 0 0) 0x7f0abc0d0ad0 con 0x7f0ac8108490 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 -- 172.21.15.63:0/75302387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 msgr2=0x55e34e8f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e8f5d90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0ac402ae10 tx=0x7f0ac40093e0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0acde00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 0x7f0ab00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 -- 172.21.15.63:0/75302387 shutdown_connections 2024-04-25T18:53:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e902f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e900650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 --2- 172.21.15.63:0/75302387 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e8f5d90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 -- 172.21.15.63:0/75302387 >> 172.21.15.63:0/75302387 conn(0x55e34e8f1ec0 msgr2=0x55e34e8f22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 -- 172.21.15.63:0/75302387 shutdown_connections 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.809+0000 7f0ad376c740 1 -- 172.21.15.63:0/75302387 wait complete. 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0acde00640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 0x7f0ab00a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0ce260 tx=0x7f0abc000bc0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 -- start start 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 Processor -- start 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 0x55e34e8fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e900b90 0x55e34e8fe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e34e903760 con 0x55e34e900b90 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e34e815310 con 0x55e34e8f59b0 2024-04-25T18:53:12.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.810+0000 7f0ad376c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e34e9038e0 con 0x55e34e8f62d0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0ad376c740 10 client.0 _ll_register_callbacks cb 0x55e34e7e3cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0ace800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e900b90 0x55e34e8fe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0ace800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 0x55e34e8fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34168/0 (socket says 172.21.15.63:34168) 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 -- 172.21.15.63:0/1316758540 learned_addr learned my addr 172.21.15.63:0/1316758540 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 -- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 msgr2=0x55e34e8fdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 0x55e34e8fdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 -- 172.21.15.63:0/1316758540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e900b90 msgr2=0x55e34e8fe5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e900b90 0x55e34e8fe5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0acd400640 1 -- 172.21.15.63:0/1316758540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e34e63a620 con 0x55e34e8f62d0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.811+0000 7f0ace800640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 0x55e34e8fdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0acd400640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800b950 tx=0x7f0ab800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c92f14963f281436 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0aab400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:53:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8003da0 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8631+0+0 (secure 0 0 0) 0x7f0ab8011440 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac4036030 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ab8005020 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0ad376c740 10 client.26023 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.812+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e34e90f9e0 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac4036030 con 0x55e34e8f62d0 2024-04-25T18:53:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:12 smithi114 ceph-mon[32988]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:12.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:12 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0ab8011640 con 0x55e34e8f62d0 2024-04-25T18:53:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 10 client.26023 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 10 client.26023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 20 client.26023 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e910f60 0x55e34e913370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 4 client.26023 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.813+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e34e86f8b0 con 0x55e34e910f60 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.814+0000 7f0ace800640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e910f60 0x55e34e913370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.814+0000 7f0ace800640 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e910f60 0x55e34e913370 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dff6134ca905717a server_cookie=eade15e7896b8e78 in_seq=0 out_seq=0 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.814+0000 7f0aab400640 10 client.26023 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0aab400640 1 -- 172.21.15.63:0/1316758540 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e34e86f8b0 con 0x55e34e910f60 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0aab400640 10 client.26023 handle_command_reply: tid=0 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 client.26023 shutdown 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 msgr2=0x55e34e8fe0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800b950 tx=0x7f0ab800be10 comp rx=0 tx=0).stop 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 shutdown_connections 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e910f60 0x55e34e913370 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e900b90 0x55e34e8fe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e8fe0a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0aab400640 0 client.26023 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 --2- 172.21.15.63:0/1316758540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e8f59b0 0x55e34e8fdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.815+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 >> 172.21.15.63:0/1316758540 conn(0x55e34e8f1ec0 msgr2=0x55e34e8f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.816+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 shutdown_connections 2024-04-25T18:53:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.816+0000 7f0ad376c740 1 -- 172.21.15.63:0/1316758540 wait complete. 2024-04-25T18:53:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.816+0000 7f0ad376c740 20 client.26023 trim_cache size 0 max 16384 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.832+0000 7f0ad376c740 1 Processor -- start 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.832+0000 7f0ad376c740 1 -- start start 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e918c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9191c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b6a70 con 0x55e34e8f62d0 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b4280 con 0x55e34e900b90 2024-04-25T18:53:12.834 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ad376c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b20f0 con 0x55e34e8f59b0 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34172/0 (socket says 172.21.15.63:34172) 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 -- 172.21.15.63:0/625534157 learned_addr learned my addr 172.21.15.63:0/625534157 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0acd400640 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e918c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 -- 172.21.15.63:0/625534157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 msgr2=0x55e34e9191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 -- 172.21.15.63:0/625534157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 msgr2=0x55e34e918c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e918c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.833+0000 7f0ace800640 1 -- 172.21.15.63:0/625534157 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e34e8f6c50 con 0x55e34e8f59b0 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.834+0000 7f0ace800640 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4004080 tx=0x7f0ac402a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=683e55f608032cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.834+0000 7f0ab6600640 1 -- 172.21.15.63:0/625534157 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac404a040 con 0x55e34e8f59b0 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.834+0000 7f0ab6600640 1 -- 172.21.15.63:0/625534157 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac4036050 con 0x55e34e8f59b0 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.834+0000 7f0ab6600640 1 -- 172.21.15.63:0/625534157 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4045440 con 0x55e34e8f59b0 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- 172.21.15.63:0/625534157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 msgr2=0x55e34e90c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4004080 tx=0x7f0ac402a8f0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- 172.21.15.63:0/625534157 shutdown_connections 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f62d0 0x55e34e918c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 --2- 172.21.15.63:0/625534157 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f59b0 0x55e34e90c620 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- 172.21.15.63:0/625534157 >> 172.21.15.63:0/625534157 conn(0x55e34e8f1ec0 msgr2=0x55e34e8f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- 172.21.15.63:0/625534157 shutdown_connections 2024-04-25T18:53:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- 172.21.15.63:0/625534157 wait complete. 2024-04-25T18:53:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 -- start start 2024-04-25T18:53:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.835+0000 7f0ad376c740 1 Processor -- start 2024-04-25T18:53:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e906230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b2740 con 0x55e34e8f59b0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b3bb0 con 0x55e34e900b90 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac81b2e10 con 0x55e34e8f62d0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ad376c740 10 client.0 _ll_register_callbacks cb 0x7f0ab8004380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ace800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ace800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47526/0 (socket says 172.21.15.63:47526) 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0ace800640 1 -- 172.21.15.63:0/952485456 learned_addr learned my addr 172.21.15.63:0/952485456 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.836+0000 7f0acd400640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e906230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 -- 172.21.15.63:0/952485456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 msgr2=0x55e34e906230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e906230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 -- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 msgr2=0x55e34e9754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 -- 172.21.15.63:0/952485456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e34e975a30 con 0x55e34e8f59b0 2024-04-25T18:53:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ace800640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4030aa0 tx=0x7f0ac40309a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c7b6d2f64b437b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0aab400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4068060 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8631+0+0 (secure 0 0 0) 0x7f0ac4030b40 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.837+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac4036050 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ac4003ba0 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 10 client.15825 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55e34e976be0 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ac4036050 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0ac4043070 con 0x55e34e8f59b0 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 10 client.15825 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 10 client.15825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 20 client.15825 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e976e00 0x55e34e979220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 4 client.15825 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:53:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.838+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e34e869320 con 0x55e34e976e00 2024-04-25T18:53:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.839+0000 7f0acd400640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e976e00 0x55e34e979220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.839+0000 7f0acd400640 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e976e00 0x55e34e979220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75e5b510a00f8972 server_cookie=cade68fd7407725c in_seq=0 out_seq=0 2024-04-25T18:53:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.839+0000 7f0aab400640 10 client.15825 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:12 smithi063 ceph-mon[28006]: pgmap v204: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:12 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0aab400640 1 -- 172.21.15.63:0/952485456 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e34e869320 con 0x55e34e976e00 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0aab400640 10 client.15825 handle_command_reply: tid=0 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 client.15825 shutdown 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 msgr2=0x55e34e905cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4030aa0 tx=0x7f0ac40309a0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 shutdown_connections 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e34e976e00 0x55e34e979220 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e34e900b90 0x55e34e9754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e34e8f62d0 0x55e34e906230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 --2- 172.21.15.63:0/952485456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34e8f59b0 0x55e34e905cf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 >> 172.21.15.63:0/952485456 conn(0x55e34e8f1ec0 msgr2=0x55e34e8fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.890+0000 7f0aab400640 0 client.15825 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.891+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 shutdown_connections 2024-04-25T18:53:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.891+0000 7f0ad376c740 1 -- 172.21.15.63:0/952485456 wait complete. 2024-04-25T18:53:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.891+0000 7f0ad376c740 20 client.15825 trim_cache size 0 max 16384 2024-04-25T18:53:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:53:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:53:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:53:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:53:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 msgr2=0x7f0ab00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 0x7f0ab00a61d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0ce260 tx=0x7f0abc000bc0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 msgr2=0x7f0ac81a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00e9f0 tx=0x7f0abc00eeb0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 shutdown_connections 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0ab00a3d10 0x7f0ab00a61d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0ac810e420 0x7f0ac819e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac8108e10 0x7f0ac81a4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 --2- 172.21.15.63:0/3865121552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac8108490 0x7f0ac81a47a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 >> 172.21.15.63:0/3865121552 conn(0x7f0ac8077d30 msgr2=0x7f0ac8078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 shutdown_connections 2024-04-25T18:53:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:12.892+0000 7f0acf200640 1 -- 172.21.15.63:0/3865121552 wait complete. 2024-04-25T18:53:13.470 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.20 MB/sec execute 192 sec latency 1294.134 ms 2024-04-25T18:53:14.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.18 MB/sec execute 193 sec latency 2294.269 ms 2024-04-25T18:53:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:14 smithi070 ceph-mon[32820]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:14 smithi070 ceph-mon[32820]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:14 smithi114 ceph-mon[32988]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:14 smithi114 ceph-mon[32988]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T18:53:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:14 smithi063 ceph-mon[28006]: pgmap v205: 97 pgs: 97 active+clean; 74 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 243 op/s 2024-04-25T18:53:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:14 smithi063 ceph-mon[28006]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T18:53:15.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.16 MB/sec execute 194 sec latency 3294.486 ms 2024-04-25T18:53:16.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.14 MB/sec execute 195 sec latency 4294.597 ms 2024-04-25T18:53:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:16 smithi070 ceph-mon[32820]: pgmap v207: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:16 smithi114 ceph-mon[32988]: pgmap v207: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:16 smithi063 ceph-mon[28006]: pgmap v207: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:17.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.12 MB/sec execute 196 sec latency 5294.692 ms 2024-04-25T18:53:18.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.09 MB/sec execute 197 sec latency 6294.853 ms 2024-04-25T18:53:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:18 smithi070 ceph-mon[32820]: pgmap v208: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:18 smithi114 ceph-mon[32988]: pgmap v208: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:18.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:18 smithi063 ceph-mon[28006]: pgmap v208: 97 pgs: 97 active+clean; 69 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 969 KiB/s wr, 155 op/s 2024-04-25T18:53:19.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.07 MB/sec execute 198 sec latency 7294.966 ms 2024-04-25T18:53:20.471 INFO:tasks.workunit.client.0.smithi063.stdout: 1 214666 4.05 MB/sec execute 199 sec latency 8295.117 ms 2024-04-25T18:53:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:20 smithi114 ceph-mon[32988]: pgmap v209: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:20 smithi063 ceph-mon[28006]: pgmap v209: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:20.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:20 smithi070 ceph-mon[32820]: pgmap v209: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:21.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 215127 4.04 MB/sec execute 200 sec latency 9010.103 ms 2024-04-25T18:53:22.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 216244 4.06 MB/sec execute 201 sec latency 44.901 ms 2024-04-25T18:53:22.755 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:22 smithi070 ceph-mon[32820]: pgmap v210: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:22.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:22 smithi114 ceph-mon[32988]: pgmap v210: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:22.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:22 smithi063 ceph-mon[28006]: pgmap v210: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:23.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 217089 4.06 MB/sec execute 202 sec latency 28.597 ms 2024-04-25T18:53:24.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 218115 4.07 MB/sec execute 203 sec latency 27.689 ms 2024-04-25T18:53:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:24 smithi114 ceph-mon[32988]: pgmap v211: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:24.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:24 smithi063 ceph-mon[28006]: pgmap v211: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:24 smithi070 ceph-mon[32820]: pgmap v211: 97 pgs: 97 active+clean; 72 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 697 KiB/s wr, 94 op/s 2024-04-25T18:53:25.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 219596 4.09 MB/sec execute 204 sec latency 23.077 ms 2024-04-25T18:53:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 -- 172.21.15.63:0/4012580308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644103830 msgr2=0x7f064410bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 --2- 172.21.15.63:0/4012580308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644103830 0x7f064410bd10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0630009900 tx=0x7f063002a480 comp rx=0 tx=0).stop 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 -- 172.21.15.63:0/4012580308 shutdown_connections 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 --2- 172.21.15.63:0/4012580308 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0644103830 0x7f064410bd10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 --2- 172.21.15.63:0/4012580308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f064410c330 0x7f064410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 --2- 172.21.15.63:0/4012580308 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 -- 172.21.15.63:0/4012580308 >> 172.21.15.63:0/4012580308 conn(0x7f06440fc770 msgr2=0x7f06440feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 -- 172.21.15.63:0/4012580308 shutdown_connections 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.038+0000 7f064b000640 1 -- 172.21.15.63:0/4012580308 wait complete. 2024-04-25T18:53:26.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.039+0000 7f064b000640 1 Processor -- start 2024-04-25T18:53:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 -- start start 2024-04-25T18:53:26.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0644103830 0x7f0644198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f064410c330 0x7f064419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0644112010 con 0x7f0644103830 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0644111e90 con 0x7f064410c330 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f064b000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0644112190 con 0x7f0644102f10 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f0649c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f0649c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43638/0 (socket says 172.21.15.63:43638) 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.040+0000 7f0649c00640 1 -- 172.21.15.63:0/891083497 learned_addr learned my addr 172.21.15.63:0/891083497 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649200640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0644103830 0x7f0644198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f064a600640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f064410c330 0x7f064419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 -- 172.21.15.63:0/891083497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f064410c330 msgr2=0x7f064419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f064410c330 0x7f064419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 -- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0644103830 msgr2=0x7f0644198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0644103830 0x7f0644198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 -- 172.21.15.63:0/891083497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064419d590 con 0x7f0644102f10 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.041+0000 7f0649c00640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.042+0000 7f0642000640 1 -- 172.21.15.63:0/891083497 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063400cd80 con 0x7f0644102f10 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.042+0000 7f0642000640 1 -- 172.21.15.63:0/891083497 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06340054d0 con 0x7f0644102f10 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.042+0000 7f0642000640 1 -- 172.21.15.63:0/891083497 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0634010690 con 0x7f0644102f10 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.042+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064419d880 con 0x7f0644102f10 2024-04-25T18:53:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.042+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06441a5110 con 0x7f0644102f10 2024-04-25T18:53:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.045+0000 7f0642000640 1 -- 172.21.15.63:0/891083497 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0634004e90 con 0x7f0644102f10 2024-04-25T18:53:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.045+0000 7f064f3de740 1 Processor -- start 2024-04-25T18:53:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.045+0000 7f064f3de740 1 -- start start 2024-04-25T18:53:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb0795a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb0798390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555cb06ec900 con 0x555cb07871f0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555cb0692ef0 con 0x555cb07868d0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064f3de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555cb06e88c0 con 0x555cb0795fa0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58688/0 (socket says 172.21.15.63:58688) 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064a600640 1 -- 172.21.15.63:0/1828837273 learned_addr learned my addr 172.21.15.63:0/1828837273 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f0649200640 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb0798390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f0649c00640 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb0795a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.046+0000 7f064a600640 1 -- 172.21.15.63:0/1828837273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 msgr2=0x555cb0798390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f064a600640 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb0798390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f064a600640 1 -- 172.21.15.63:0/1828837273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 msgr2=0x555cb0795a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f064a600640 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb0795a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f064a600640 1 -- 172.21.15.63:0/1828837273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cb03ff210 con 0x555cb07868d0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f064a600640 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f063c009f70 tx=0x7f063c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e330a9c712f5a3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f063aa00640 1 -- 172.21.15.63:0/1828837273 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c016070 con 0x555cb07868d0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f063aa00640 1 -- 172.21.15.63:0/1828837273 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c004d20 con 0x555cb07868d0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.047+0000 7f063aa00640 1 -- 172.21.15.63:0/1828837273 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c0127d0 con 0x555cb07868d0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f0642000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 0x7f06280a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 -- 172.21.15.63:0/1828837273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 msgr2=0x555cb0786cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f063c009f70 tx=0x7f063c010800 comp rx=0 tx=0).stop 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f0642000640 1 -- 172.21.15.63:0/891083497 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f06340d0c10 con 0x7f0644102f10 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f0649c00640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 0x7f06280a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 -- 172.21.15.63:0/1828837273 shutdown_connections 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb0798390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb0795a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 --2- 172.21.15.63:0/1828837273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb0786cb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 -- 172.21.15.63:0/1828837273 >> 172.21.15.63:0/1828837273 conn(0x555cb0782ec0 msgr2=0x555cb07832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 -- 172.21.15.63:0/1828837273 shutdown_connections 2024-04-25T18:53:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.048+0000 7f064f3de740 1 -- 172.21.15.63:0/1828837273 wait complete. 2024-04-25T18:53:26.056 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.049+0000 7f0649c00640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 0x7f06280a6250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x555cb07894d0 tx=0x7f0634004a90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.049+0000 7f064f3de740 1 -- start start 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.049+0000 7f064f3de740 1 Processor -- start 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.049+0000 7f064f3de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb0795a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb0792f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb07934b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555cb0798bd0 con 0x555cb07868d0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x555cb06a6310 con 0x555cb0795fa0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x555cb0798d50 con 0x555cb07871f0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064f3de740 10 client.0 _ll_register_callbacks cb 0x555cb0674cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064a600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb0795a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb0792f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064a600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb0795a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57318/0 (socket says 172.21.15.63:57318) 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064a600640 1 -- 172.21.15.63:0/2902104519 learned_addr learned my addr 172.21.15.63:0/2902104519 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f064a600640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb07934b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.057 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 -- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 msgr2=0x555cb07934b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb07934b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 -- 172.21.15.63:0/2902104519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 msgr2=0x555cb0795a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb0795a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.050+0000 7f0649200640 1 -- 172.21.15.63:0/2902104519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cb04cb620 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f0649200640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb0792f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f063002aa70 tx=0x7f06300043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f486f00f3bdd1ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f062f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630035040 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f063c004d20 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f0630005450 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0630005650 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.051+0000 7f064f3de740 10 client.26041 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555cb07a92d0 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063c004d20 con 0x555cb07871f0 2024-04-25T18:53:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f0630003940 con 0x555cb07871f0 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 10 client.26041 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 10 client.26041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 20 client.26041 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb07aa790 0x555cb07acb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 4 client.26041 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cb07008b0 con 0x555cb07aa790 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.052+0000 7f064a600640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb07aa790 0x555cb07acb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.053+0000 7f064a600640 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb07aa790 0x555cb07acb80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ce9f9a0965feada server_cookie=ff281bc6c7d2b463 in_seq=0 out_seq=0 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.053+0000 7f062f400640 10 client.26041 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f062f400640 1 -- 172.21.15.63:0/2902104519 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555cb07008b0 con 0x555cb07aa790 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f062f400640 10 client.26041 handle_command_reply: tid=0 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f064f3de740 1 client.26041 shutdown 2024-04-25T18:53:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 msgr2=0x555cb0792f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb0792f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f063002aa70 tx=0x7f06300043b0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 shutdown_connections 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.054+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb07aa790 0x555cb07acb80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb07934b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb0792f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 --2- 172.21.15.63:0/2902104519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb0795a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f062f400640 0 client.26041 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 >> 172.21.15.63:0/2902104519 conn(0x555cb0782ec0 msgr2=0x555cb0788d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 shutdown_connections 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 1 -- 172.21.15.63:0/2902104519 wait complete. 2024-04-25T18:53:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.055+0000 7f064f3de740 20 client.26041 trim_cache size 0 max 16384 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.071+0000 7f064f3de740 1 Processor -- start 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 -- start start 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb07b2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb0791840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06441ae230 con 0x555cb07868d0 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06441aba40 con 0x555cb0795fa0 2024-04-25T18:53:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064f3de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06441a98b0 con 0x555cb07871f0 2024-04-25T18:53:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f0649200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f0649200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43662/0 (socket says 172.21.15.63:43662) 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f0649200640 1 -- 172.21.15.63:0/70806352 learned_addr learned my addr 172.21.15.63:0/70806352 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064a600640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb07b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f064a600640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb0791840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.072+0000 7f0649200640 1 -- 172.21.15.63:0/70806352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 msgr2=0x555cb0791840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f0649200640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb0791840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f0649200640 1 -- 172.21.15.63:0/70806352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 msgr2=0x555cb07b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f0649200640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb07b2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f0649200640 1 -- 172.21.15.63:0/70806352 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cb0787020 con 0x555cb07871f0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f064a600640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb0791840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f0649200640 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f063002a9a0 tx=0x7f0630001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e60f2d7a48157f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f063a000640 1 -- 172.21.15.63:0/70806352 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630035070 con 0x555cb07871f0 2024-04-25T18:53:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f063a000640 1 -- 172.21.15.63:0/70806352 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0630041070 con 0x555cb07871f0 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.073+0000 7f063a000640 1 -- 172.21.15.63:0/70806352 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0630052070 con 0x555cb07871f0 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 -- 172.21.15.63:0/70806352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 msgr2=0x555cb07b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f063002a9a0 tx=0x7f0630001e70 comp rx=0 tx=0).stop 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 -- 172.21.15.63:0/70806352 shutdown_connections 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb0795fa0 0x555cb0791840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb07871f0 0x555cb07b2990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 --2- 172.21.15.63:0/70806352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07868d0 0x555cb07b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 -- 172.21.15.63:0/70806352 >> 172.21.15.63:0/70806352 conn(0x555cb0782ec0 msgr2=0x555cb07832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 -- 172.21.15.63:0/70806352 shutdown_connections 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.074+0000 7f064f3de740 1 -- 172.21.15.63:0/70806352 wait complete. 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 -- start start 2024-04-25T18:53:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 Processor -- start 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb07a4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb07a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f06441a9f00 con 0x555cb07871f0 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06441ab370 con 0x555cb07868d0 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06441aa5d0 con 0x555cb0795fa0 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064f3de740 10 client.0 _ll_register_callbacks cb 0x7f0630004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.075+0000 7f064a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f0649200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb07a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58696/0 (socket says 172.21.15.63:58696) 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 -- 172.21.15.63:0/3470262794 learned_addr learned my addr 172.21.15.63:0/3470262794 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb07a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 -- 172.21.15.63:0/3470262794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 msgr2=0x555cb07a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb07a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 -- 172.21.15.63:0/3470262794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 msgr2=0x555cb07a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb07a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 -- 172.21.15.63:0/3470262794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cb07a5300 con 0x555cb07868d0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064a600640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f063c010d00 tx=0x7f063c014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f019e59aa7fe36a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f062f400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063c016050 con 0x555cb07868d0 2024-04-25T18:53:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0630041070 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.076+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f063c02d660 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f063c03e050 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 10 client.26024 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555cb0810cf0 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0630041070 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7f063c00f040 con 0x555cb07868d0 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 10 client.26024 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 20 client.26024 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb0811280 0x555cb08136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 4 client.26024 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cb06fa320 con 0x555cb0811280 2024-04-25T18:53:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.077+0000 7f0649200640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb0811280 0x555cb08136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.078+0000 7f0649200640 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb0811280 0x555cb08136a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44418cbcc5f87099 server_cookie=cba8af13305cb5e2 in_seq=0 out_seq=0 2024-04-25T18:53:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.078+0000 7f062f400640 10 client.26024 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.117+0000 7f062f400640 1 -- 172.21.15.63:0/3470262794 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cb06fa320 con 0x555cb0811280 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.117+0000 7f062f400640 10 client.26024 handle_command_reply: tid=0 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.117+0000 7f064f3de740 1 client.26024 shutdown 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 msgr2=0x555cb07a7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f063c010d00 tx=0x7f063c014040 comp rx=0 tx=0).stop 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 shutdown_connections 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x555cb0811280 0x555cb08136a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x555cb0795fa0 0x555cb07a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555cb07871f0 0x555cb07a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 --2- 172.21.15.63:0/3470262794 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x555cb07868d0 0x555cb07a7170 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 >> 172.21.15.63:0/3470262794 conn(0x555cb0782ec0 msgr2=0x555cb0781790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f062f400640 0 client.26024 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 shutdown_connections 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 1 -- 172.21.15.63:0/3470262794 wait complete. 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.118+0000 7f064f3de740 20 client.26024 trim_cache size 0 max 16384 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:53:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:53:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:53:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 msgr2=0x7f06280a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 0x7f06280a6250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x555cb07894d0 tx=0x7f0634004a90 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 msgr2=0x7f06441985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f063400e9f0 tx=0x7f063400edf0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 shutdown_connections 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06280a3d90 0x7f06280a6250 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f064410c330 0x7f064419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0644103830 0x7f0644198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 --2- 172.21.15.63:0/891083497 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0644102f10 0x7f06441985f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.119+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 >> 172.21.15.63:0/891083497 conn(0x7f06440fc770 msgr2=0x7f06440fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.120+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 shutdown_connections 2024-04-25T18:53:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:26.120+0000 7f064b000640 1 -- 172.21.15.63:0/891083497 wait complete. 2024-04-25T18:53:26.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 220687 4.10 MB/sec execute 205 sec latency 24.231 ms 2024-04-25T18:53:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:26 smithi114 ceph-mon[32988]: pgmap v212: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:53:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:26 smithi063 ceph-mon[28006]: pgmap v212: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:53:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:26 smithi070 ceph-mon[32820]: pgmap v212: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:53:27.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 221614 4.10 MB/sec execute 206 sec latency 22.605 ms 2024-04-25T18:53:28.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 222986 4.12 MB/sec execute 207 sec latency 85.892 ms 2024-04-25T18:53:28.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:28 smithi114 ceph-mon[32988]: pgmap v213: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 966 KiB/s wr, 61 op/s 2024-04-25T18:53:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:28 smithi063 ceph-mon[28006]: pgmap v213: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 966 KiB/s wr, 61 op/s 2024-04-25T18:53:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:28 smithi070 ceph-mon[32820]: pgmap v213: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 966 KiB/s wr, 61 op/s 2024-04-25T18:53:29.472 INFO:tasks.workunit.client.0.smithi063.stdout: 1 224224 4.13 MB/sec execute 208 sec latency 21.366 ms 2024-04-25T18:53:30.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 225134 4.13 MB/sec execute 209 sec latency 28.905 ms 2024-04-25T18:53:30.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:30 smithi114 ceph-mon[32988]: pgmap v214: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-04-25T18:53:30.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:30 smithi063 ceph-mon[28006]: pgmap v214: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-04-25T18:53:30.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:30 smithi070 ceph-mon[32820]: pgmap v214: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 147 op/s 2024-04-25T18:53:31.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 226391 4.14 MB/sec execute 210 sec latency 22.358 ms 2024-04-25T18:53:32.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 227581 4.16 MB/sec execute 211 sec latency 12.995 ms 2024-04-25T18:53:32.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:32 smithi114 ceph-mon[32988]: pgmap v215: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:32 smithi063 ceph-mon[28006]: pgmap v215: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:32.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:32 smithi070 ceph-mon[32820]: pgmap v215: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:33.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 228715 4.17 MB/sec execute 212 sec latency 26.719 ms 2024-04-25T18:53:34.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 230418 4.19 MB/sec execute 213 sec latency 23.814 ms 2024-04-25T18:53:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:34 smithi114 ceph-mon[32988]: pgmap v216: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:34.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:34 smithi063 ceph-mon[28006]: pgmap v216: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:34.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:34 smithi070 ceph-mon[32820]: pgmap v216: 97 pgs: 97 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-25T18:53:35.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 230418 4.17 MB/sec execute 214 sec latency 1003.018 ms 2024-04-25T18:53:36.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 231072 4.17 MB/sec execute 215 sec latency 1409.007 ms 2024-04-25T18:53:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:36 smithi114 ceph-mon[32988]: pgmap v217: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-25T18:53:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:36 smithi063 ceph-mon[28006]: pgmap v217: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-25T18:53:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:36 smithi070 ceph-mon[32820]: pgmap v217: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-25T18:53:37.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 232296 4.18 MB/sec execute 216 sec latency 22.585 ms 2024-04-25T18:53:38.473 INFO:tasks.workunit.client.0.smithi063.stdout: 1 233967 4.20 MB/sec execute 217 sec latency 42.720 ms 2024-04-25T18:53:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:38 smithi114 ceph-mon[32988]: pgmap v218: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-25T18:53:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:38 smithi063 ceph-mon[28006]: pgmap v218: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-25T18:53:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:38 smithi070 ceph-mon[32820]: pgmap v218: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-25T18:53:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.270+0000 7fcd9be00640 1 -- 172.21.15.63:0/1214172713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c10e420 msgr2=0x7fcd9c10e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.270+0000 7fcd9be00640 1 --2- 172.21.15.63:0/1214172713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c10e420 0x7fcd9c10e800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009cd0 tx=0x7fcd8c02a8f0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 -- 172.21.15.63:0/1214172713 shutdown_connections 2024-04-25T18:53:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 --2- 172.21.15.63:0/1214172713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 --2- 172.21.15.63:0/1214172713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c108490 0x7fcd9c1088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 --2- 172.21.15.63:0/1214172713 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c10e420 0x7fcd9c10e800 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 -- 172.21.15.63:0/1214172713 >> 172.21.15.63:0/1214172713 conn(0x7fcd9c077d30 msgr2=0x7fcd9c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 -- 172.21.15.63:0/1214172713 shutdown_connections 2024-04-25T18:53:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.271+0000 7fcd9be00640 1 -- 172.21.15.63:0/1214172713 wait complete. 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.272+0000 7fcd9be00640 1 Processor -- start 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.272+0000 7fcd9be00640 1 -- start start 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c108490 0x7fcd9c1a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c10e420 0x7fcd9c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c118650 con 0x7fcd9c10e420 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1184d0 con 0x7fcd9c108e10 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9be00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1187d0 con 0x7fcd9c108490 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46300/0 (socket says 172.21.15.63:46300) 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 -- 172.21.15.63:0/2405771209 learned_addr learned my addr 172.21.15.63:0/2405771209 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c108490 0x7fcd9c1a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd9b400640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c10e420 0x7fcd9c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 -- 172.21.15.63:0/2405771209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c108490 msgr2=0x7fcd9c1a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c108490 0x7fcd9c1a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 -- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c10e420 msgr2=0x7fcd9c19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c10e420 0x7fcd9c19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.273+0000 7fcd92000640 1 -- 172.21.15.63:0/2405771209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9c19f130 con 0x7fcd9c108e10 2024-04-25T18:53:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd92000640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400d950 tx=0x7fcd8400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd93e00640 1 -- 172.21.15.63:0/2405771209 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84017070 con 0x7fcd9c108e10 2024-04-25T18:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd93e00640 1 -- 172.21.15.63:0/2405771209 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd84009c30 con 0x7fcd9c108e10 2024-04-25T18:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd93e00640 1 -- 172.21.15.63:0/2405771209 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400f620 con 0x7fcd9c108e10 2024-04-25T18:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9c19f420 con 0x7fcd9c108e10 2024-04-25T18:53:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.274+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd9c1ab6c0 con 0x7fcd9c108e10 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcd93e00640 1 -- 172.21.15.63:0/2405771209 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcd84010460 con 0x7fcd9c108e10 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 Processor -- start 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 -- start start 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a67f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5580b99c0900 con 0x5580b9a65b90 2024-04-25T18:53:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580b9966ef0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.278+0000 7fcda43d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5580b99bc8c0 con 0x5580b9a5a9b0 2024-04-25T18:53:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56282/0 (socket says 172.21.15.63:56282) 2024-04-25T18:53:39.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46314/0 (socket says 172.21.15.63:46314) 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9b400640 1 -- 172.21.15.63:0/113635104 learned_addr learned my addr 172.21.15.63:0/113635104 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd92000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a67f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 -- 172.21.15.63:0/113635104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 msgr2=0x5580b9a5ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 -- 172.21.15.63:0/113635104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 msgr2=0x5580b9a67f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a67f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 -- 172.21.15.63:0/113635104 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b96d3210 con 0x5580b9a5b2d0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9b400640 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.279+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009ca0 tx=0x7fcd8c030fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac3228617143214 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd8b400640 1 -- 172.21.15.63:0/113635104 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c030800 con 0x5580b9a5b2d0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd8b400640 1 -- 172.21.15.63:0/113635104 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd8c036030 con 0x5580b9a5b2d0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd8b400640 1 -- 172.21.15.63:0/113635104 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0043c0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd93e00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 0x7fcd800a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd93e00640 1 -- 172.21.15.63:0/2405771209 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fcd840d0500 con 0x7fcd9c108e10 2024-04-25T18:53:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcd9b400640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 0x7fcd800a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcda43d5740 1 -- 172.21.15.63:0/113635104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 msgr2=0x5580b9a65650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.280+0000 7fcda43d5740 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009ca0 tx=0x7fcd8c030fb0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 -- 172.21.15.63:0/113635104 shutdown_connections 2024-04-25T18:53:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a65650 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a67f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 --2- 172.21.15.63:0/113635104 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 -- 172.21.15.63:0/113635104 >> 172.21.15.63:0/113635104 conn(0x5580b9a56ec0 msgr2=0x5580b9a572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 -- 172.21.15.63:0/113635104 shutdown_connections 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 -- 172.21.15.63:0/113635104 wait complete. 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcd9b400640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 0x7fcd800a61d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009e80 tx=0x7fcd94005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 -- start start 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.281+0000 7fcda43d5740 1 Processor -- start 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9a62b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a65b90 0x5580b9a635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5580b9a68760 con 0x5580b9a5a9b0 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5580b997a310 con 0x5580b9a65b90 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5580b9a688e0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcda43d5740 10 client.0 _ll_register_callbacks cb 0x5580b9948cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd9b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a65b90 0x5580b9a635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56290/0 (socket says 172.21.15.63:56290) 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 -- 172.21.15.63:0/1749196062 learned_addr learned my addr 172.21.15.63:0/1749196062 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9a62b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 -- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a65b90 msgr2=0x5580b9a635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a65b90 0x5580b9a635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 -- 172.21.15.63:0/1749196062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 msgr2=0x5580b9a62b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9a62b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.282+0000 7fcd92000640 1 -- 172.21.15.63:0/1749196062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b979f620 con 0x5580b9a5b2d0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9a62b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd92000640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcd840cdc10 tx=0x7fcd84005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da6f4a459730bac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84020070 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fcd8409b850 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd8c036030 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcd8409bf40 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcda43d5740 10 client.26059 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.283+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5580b9a749e0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd8c036030 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcd84010750 con 0x5580b9a5b2d0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 10 client.26059 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 10 client.26059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 20 client.26059 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9a75ea0 0x5580b9a782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 4 client.26059 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b99d48b0 con 0x5580b9a75ea0 2024-04-25T18:53:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.284+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9a75ea0 0x5580b9a782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.285+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9a75ea0 0x5580b9a782f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec413235a7a6ca0 server_cookie=fbab4cd1e7d77efd in_seq=0 out_seq=0 2024-04-25T18:53:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.285+0000 7fcd7fe00640 10 client.26059 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1749196062 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580b99d48b0 con 0x5580b9a75ea0 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcd7fe00640 10 client.26059 handle_command_reply: tid=0 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 client.26059 shutdown 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 msgr2=0x5580b9a63080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcd840cdc10 tx=0x7fcd84005f70 comp rx=0 tx=0).stop 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 shutdown_connections 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9a75ea0 0x5580b9a782f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a65b90 0x5580b9a635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5b2d0 0x5580b9a63080 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1749196062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9a62b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcd7fe00640 0 client.26059 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 >> 172.21.15.63:0/1749196062 conn(0x5580b9a56ec0 msgr2=0x5580b9a5cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 shutdown_connections 2024-04-25T18:53:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 1 -- 172.21.15.63:0/1749196062 wait complete. 2024-04-25T18:53:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.286+0000 7fcda43d5740 20 client.26059 trim_cache size 0 max 16384 2024-04-25T18:53:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.304+0000 7fcda43d5740 1 Processor -- start 2024-04-25T18:53:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.304+0000 7fcda43d5740 1 -- start start 2024-04-25T18:53:39.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a71620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a7dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b6a90 con 0x5580b9a65b90 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b42a0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcda43d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b2110 con 0x5580b9a5a9b0 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46340/0 (socket says 172.21.15.63:46340) 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 -- 172.21.15.63:0/1547382366 learned_addr learned my addr 172.21.15.63:0/1547382366 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd9b400640 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a7dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a71620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 -- 172.21.15.63:0/1547382366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 msgr2=0x5580b9a71620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a71620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 -- 172.21.15.63:0/1547382366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 msgr2=0x5580b9a7dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a7dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.305+0000 7fcd92000640 1 -- 172.21.15.63:0/1547382366 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b9a56ce0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcd92000640 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcd84098860 tx=0x7fcd840d0780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcbf3685a842c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcd8aa00640 1 -- 172.21.15.63:0/1547382366 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd840b5070 con 0x5580b9a5b2d0 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcd8aa00640 1 -- 172.21.15.63:0/1547382366 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd84009de0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcd8aa00640 1 -- 172.21.15.63:0/1547382366 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84020040 con 0x5580b9a5b2d0 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 -- 172.21.15.63:0/1547382366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 msgr2=0x5580b9a567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcd84098860 tx=0x7fcd840d0780 comp rx=0 tx=0).stop 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 -- 172.21.15.63:0/1547382366 shutdown_connections 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9a567a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a65b90 0x5580b9a7dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1547382366 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a5a9b0 0x5580b9a71620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.306+0000 7fcda43d5740 1 -- 172.21.15.63:0/1547382366 >> 172.21.15.63:0/1547382366 conn(0x5580b9a56ec0 msgr2=0x5580b9a5de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.307+0000 7fcda43d5740 1 -- 172.21.15.63:0/1547382366 shutdown_connections 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.307+0000 7fcda43d5740 1 -- 172.21.15.63:0/1547382366 wait complete. 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.307+0000 7fcda43d5740 1 -- start start 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.307+0000 7fcda43d5740 1 Processor -- start 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9ad4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a65b90 0x5580b9a6ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b2760 con 0x5580b9a5a9b0 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b3bd0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c1b2e30 con 0x5580b9a65b90 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcda43d5740 10 client.0 _ll_register_callbacks cb 0x7fcd8400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd9aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9ad4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46344/0 (socket says 172.21.15.63:46344) 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 -- 172.21.15.63:0/1341652156 learned_addr learned my addr 172.21.15.63:0/1341652156 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd9b400640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a65b90 0x5580b9a6ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 -- 172.21.15.63:0/1341652156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a65b90 msgr2=0x5580b9a6ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a65b90 0x5580b9a6ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 -- 172.21.15.63:0/1341652156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 msgr2=0x5580b9ad4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9ad4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd92000640 1 -- 172.21.15.63:0/1341652156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b9a6b550 con 0x5580b9a5b2d0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.308+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9ad4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd92000640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcd84000c00 tx=0x7fcd840d3e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6026a3f891153a34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84020070 con 0x5580b9a5b2d0 2024-04-25T18:53:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fcd84098150 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd84009de0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcd840b5070 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcda43d5740 10 client.26042 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5580b9adc030 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd84009de0 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.309+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fcd840a0180 con 0x5580b9a5b2d0 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 10 client.26042 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 10 client.26042 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 20 client.26042 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9adc720 0x5580b9adeb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 4 client.26042 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580b99ce320 con 0x5580b9adc720 2024-04-25T18:53:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9adc720 0x5580b9adeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcd9aa00640 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9adc720 0x5580b9adeb40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a2245caf7de6f4 server_cookie=680b090c38d3746f in_seq=0 out_seq=0 2024-04-25T18:53:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.310+0000 7fcd7fe00640 10 client.26042 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.361+0000 7fcd7fe00640 1 -- 172.21.15.63:0/1341652156 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580b99ce320 con 0x5580b9adc720 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.361+0000 7fcd7fe00640 10 client.26042 handle_command_reply: tid=0 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.361+0000 7fcda43d5740 1 client.26042 shutdown 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 msgr2=0x5580b9ad4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcd84000c00 tx=0x7fcd840d3e10 comp rx=0 tx=0).stop 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 shutdown_connections 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5580b9adc720 0x5580b9adeb40 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5580b9a5b2d0 0x5580b9ad4da0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5580b9a65b90 0x5580b9a6ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 --2- 172.21.15.63:0/1341652156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580b9a5a9b0 0x5580b9ad4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 >> 172.21.15.63:0/1341652156 conn(0x5580b9a56ec0 msgr2=0x5580b9a5c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcd7fe00640 0 client.26042 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 shutdown_connections 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 1 -- 172.21.15.63:0/1341652156 wait complete. 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.362+0000 7fcda43d5740 20 client.26042 trim_cache size 0 max 16384 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:53:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 msgr2=0x7fcd800a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 0x7fcd800a61d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009e80 tx=0x7fcd94005f90 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 msgr2=0x7fcd9c1a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400d950 tx=0x7fcd8400de10 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 shutdown_connections 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcd800a3d10 0x7fcd800a61d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd9c108e10 0x7fcd9c1a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd9c10e420 0x7fcd9c19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 --2- 172.21.15.63:0/2405771209 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcd9c108490 0x7fcd9c1a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 >> 172.21.15.63:0/2405771209 conn(0x7fcd9c077d30 msgr2=0x7fcd9c078e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 shutdown_connections 2024-04-25T18:53:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:39.363+0000 7fcd9be00640 1 -- 172.21.15.63:0/2405771209 wait complete. 2024-04-25T18:53:39.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 235006 4.20 MB/sec execute 218 sec latency 37.111 ms 2024-04-25T18:53:40.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 235842 4.21 MB/sec execute 219 sec latency 57.769 ms 2024-04-25T18:53:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:40 smithi114 ceph-mon[32988]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 212 op/s 2024-04-25T18:53:40.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:40 smithi063 ceph-mon[28006]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 212 op/s 2024-04-25T18:53:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:40 smithi070 ceph-mon[32820]: pgmap v219: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 212 op/s 2024-04-25T18:53:41.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 237314 4.22 MB/sec execute 220 sec latency 20.309 ms 2024-04-25T18:53:42.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 238497 4.23 MB/sec execute 221 sec latency 16.747 ms 2024-04-25T18:53:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:42 smithi114 ceph-mon[32988]: pgmap v220: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:42 smithi063 ceph-mon[28006]: pgmap v220: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:42 smithi070 ceph-mon[32820]: pgmap v220: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:43.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 240073 4.24 MB/sec execute 222 sec latency 53.660 ms 2024-04-25T18:53:44.474 INFO:tasks.workunit.client.0.smithi063.stdout: 1 241566 4.28 MB/sec execute 223 sec latency 11.088 ms 2024-04-25T18:53:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:44 smithi114 ceph-mon[32988]: pgmap v221: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:44 smithi063 ceph-mon[28006]: pgmap v221: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:44 smithi070 ceph-mon[32820]: pgmap v221: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.4 MiB/s wr, 127 op/s 2024-04-25T18:53:45.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 242706 4.28 MB/sec execute 224 sec latency 22.467 ms 2024-04-25T18:53:46.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 243635 4.27 MB/sec execute 225 sec latency 20.273 ms 2024-04-25T18:53:46.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:46 smithi114 ceph-mon[32988]: pgmap v222: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-25T18:53:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:46 smithi063 ceph-mon[28006]: pgmap v222: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-25T18:53:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:46 smithi070 ceph-mon[32820]: pgmap v222: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.2 MiB/s wr, 235 op/s 2024-04-25T18:53:47.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 245055 4.31 MB/sec execute 226 sec latency 25.079 ms 2024-04-25T18:53:48.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 246066 4.30 MB/sec execute 227 sec latency 19.527 ms 2024-04-25T18:53:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:48 smithi114 ceph-mon[32988]: pgmap v223: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-25T18:53:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:48 smithi063 ceph-mon[28006]: pgmap v223: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-25T18:53:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:48 smithi070 ceph-mon[32820]: pgmap v223: 97 pgs: 97 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-25T18:53:49.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 247554 4.32 MB/sec execute 228 sec latency 22.367 ms 2024-04-25T18:53:49.950 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.4 2024-04-25T18:53:49.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:53:50.451 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:53:50.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 248836 4.33 MB/sec execute 229 sec latency 13.127 ms 2024-04-25T18:53:50.676 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:50 smithi063 ceph-mon[28006]: pgmap v224: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 225 op/s 2024-04-25T18:53:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:50 smithi114 ceph-mon[32988]: pgmap v224: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 225 op/s 2024-04-25T18:53:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:50 smithi070 ceph-mon[32820]: pgmap v224: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 225 op/s 2024-04-25T18:53:51.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 249888 4.34 MB/sec execute 230 sec latency 20.757 ms 2024-04-25T18:53:52.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 251341 4.35 MB/sec execute 231 sec latency 30.777 ms 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 -- 172.21.15.63:0/3101424118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd544109470 msgr2=0x7fd544111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 --2- 172.21.15.63:0/3101424118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd544109470 0x7fd544111040 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009ca0 tx=0x7fd53c02a8c0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 -- 172.21.15.63:0/3101424118 shutdown_connections 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 --2- 172.21.15.63:0/3101424118 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd544109470 0x7fd544111040 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 --2- 172.21.15.63:0/3101424118 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd544104d50 0x7fd544108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 --2- 172.21.15.63:0/3101424118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5441043a0 0x7fd544104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 -- 172.21.15.63:0/3101424118 >> 172.21.15.63:0/3101424118 conn(0x7fd5440ffc40 msgr2=0x7fd544102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 -- 172.21.15.63:0/3101424118 shutdown_connections 2024-04-25T18:53:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.513+0000 7fd54a400640 1 -- 172.21.15.63:0/3101424118 wait complete. 2024-04-25T18:53:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.514+0000 7fd54a400640 1 Processor -- start 2024-04-25T18:53:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.514+0000 7fd54a400640 1 -- start start 2024-04-25T18:53:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd544109470 0x7fd5441a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441143c0 con 0x7fd544109470 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd544114240 con 0x7fd54419a6c0 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd54a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd544114540 con 0x7fd5441043a0 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53634/0 (socket says 172.21.15.63:53634) 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 -- 172.21.15.63:0/1980872135 learned_addr learned my addr 172.21.15.63:0/1980872135 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:60842/0 (socket says 172.21.15.63:60842) 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd53be00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd544109470 0x7fd5441a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 -- 172.21.15.63:0/1980872135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 msgr2=0x7fd5441a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 -- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd544109470 msgr2=0x7fd5441a0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd544109470 0x7fd5441a0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543e00640 1 -- 172.21.15.63:0/1980872135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54419b2c0 con 0x7fd54419a6c0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.515+0000 7fd543400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd543e00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005cc0 tx=0x7fd53c031700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd540c00640 1 -- 172.21.15.63:0/1980872135 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c003b70 con 0x7fd54419a6c0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54419b550 con 0x7fd54419a6c0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd540c00640 1 -- 172.21.15.63:0/1980872135 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53c0318f0 con 0x7fd54419a6c0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd540c00640 1 -- 172.21.15.63:0/1980872135 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c02f950 con 0x7fd54419a6c0 2024-04-25T18:53:52.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.516+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441a76f0 con 0x7fd54419a6c0 2024-04-25T18:53:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.519+0000 7fd54d19c740 1 Processor -- start 2024-04-25T18:53:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.520+0000 7fd540c00640 1 -- 172.21.15.63:0/1980872135 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd53c02fb50 con 0x7fd54419a6c0 2024-04-25T18:53:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.520+0000 7fd54d19c740 1 -- start start 2024-04-25T18:53:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.520+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b6ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.520+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.520+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 0x55fb60b77df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd54d19c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fb60ad0900 con 0x55fb60b75a00 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd54d19c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fb60a76ef0 con 0x55fb60b6b140 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd54d19c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb60acc8c0 con 0x55fb60b6a820 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd53be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b6ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53638/0 (socket says 172.21.15.63:53638) 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 -- 172.21.15.63:0/1560791784 learned_addr learned my addr 172.21.15.63:0/1560791784 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd53be00640 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 0x55fb60b77df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 -- 172.21.15.63:0/1560791784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 msgr2=0x55fb60b6ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b6ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 -- 172.21.15.63:0/1560791784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 msgr2=0x55fb60b77df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 0x55fb60b77df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd543400640 1 -- 172.21.15.63:0/1560791784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb607e3210 con 0x55fb60b6b140 2024-04-25T18:53:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.521+0000 7fd53be00640 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 0x55fb60b77df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:53:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd543400640 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd5340115f0 tx=0x7fd534011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79f5ce4f89a216d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd540c00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 0x7fd5280a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd538c00640 1 -- 172.21.15.63:0/1560791784 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53401e070 con 0x55fb60b6b140 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd538c00640 1 -- 172.21.15.63:0/1560791784 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53400faf0 con 0x55fb60b6b140 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd538c00640 1 -- 172.21.15.63:0/1560791784 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534019bc0 con 0x55fb60b6b140 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd540c00640 1 -- 172.21.15.63:0/1980872135 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fd53c0f1a00 con 0x7fd54419a6c0 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.522+0000 7fd53be00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 0x7fd5280a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- 172.21.15.63:0/1560791784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 msgr2=0x55fb60b754c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd5340115f0 tx=0x7fd534011ab0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- 172.21.15.63:0/1560791784 shutdown_connections 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6b140 0x55fb60b754c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b75a00 0x55fb60b77df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 --2- 172.21.15.63:0/1560791784 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b6ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- 172.21.15.63:0/1560791784 >> 172.21.15.63:0/1560791784 conn(0x55fb60b66d30 msgr2=0x55fb60b67110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd53be00640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 0x7fd5280a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55fb60b68150 tx=0x7fd52c013040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- 172.21.15.63:0/1560791784 shutdown_connections 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- 172.21.15.63:0/1560791784 wait complete. 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.523+0000 7fd54d19c740 1 -- start start 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 Processor -- start 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b72970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b72eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fb60b78630 con 0x55fb60b6b140 2024-04-25T18:53:52.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fb60a8a310 con 0x55fb60b75a00 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fb60b787b0 con 0x55fb60b6a820 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd54d19c740 10 client.0 _ll_register_callbacks cb 0x55fb60a58cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd543e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd543e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53648/0 (socket says 172.21.15.63:53648) 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd543e00640 1 -- 172.21.15.63:0/2684797368 learned_addr learned my addr 172.21.15.63:0/2684797368 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd543400640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b72970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.524+0000 7fd53be00640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b72eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 -- 172.21.15.63:0/2684797368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 msgr2=0x55fb60b72970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b72970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 -- 172.21.15.63:0/2684797368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 msgr2=0x55fb60b72eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b72eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 -- 172.21.15.63:0/2684797368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb608af620 con 0x55fb60b75a00 2024-04-25T18:53:52.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543400640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b72970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd543e00640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0bd9a0 tx=0x7fd53c0be110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f24b37aa89daed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd531600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0bda10 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fd53c0ef330 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.525+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53400faf0 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd53c0c1dc0 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 10 client.26054 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb60b78630 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53400faf0 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd53c0dcbd0 con 0x55fb60b75a00 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 10 client.26054 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 20 client.26054 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:52.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60b85d60 0x55fb60b881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 4 client.26054 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:53:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb60ae48b0 con 0x55fb60b85d60 2024-04-25T18:53:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.526+0000 7fd543400640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60b85d60 0x55fb60b881c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.527+0000 7fd543400640 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60b85d60 0x55fb60b881c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c25b5126e8ef807a server_cookie=7183382f75b048e9 in_seq=0 out_seq=0 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.527+0000 7fd531600640 10 client.26054 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd531600640 1 -- 172.21.15.63:0/2684797368 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb60ae48b0 con 0x55fb60b85d60 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd531600640 10 client.26054 handle_command_reply: tid=0 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 client.26054 shutdown 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 msgr2=0x55fb60b733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0bd9a0 tx=0x7fd53c0be110 comp rx=0 tx=0).stop 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 shutdown_connections 2024-04-25T18:53:52.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60b85d60 0x55fb60b881c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b733f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b72eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2684797368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6a820 0x55fb60b72970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd531600640 0 client.26054 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 >> 172.21.15.63:0/2684797368 conn(0x55fb60b66d30 msgr2=0x55fb60b6ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 shutdown_connections 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 1 -- 172.21.15.63:0/2684797368 wait complete. 2024-04-25T18:53:52.534 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.528+0000 7fd54d19c740 20 client.26054 trim_cache size 0 max 16384 2024-04-25T18:53:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 Processor -- start 2024-04-25T18:53:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 -- start start 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b6c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b75a00 0x55fb60b8dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441b2770 con 0x55fb60b6b140 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441aff80 con 0x55fb60b6a820 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd54d19c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441addf0 con 0x55fb60b75a00 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd543400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd53be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b6c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd543400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53652/0 (socket says 172.21.15.63:53652) 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd543e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b75a00 0x55fb60b8dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.547+0000 7fd543400640 1 -- 172.21.15.63:0/2119093778 learned_addr learned my addr 172.21.15.63:0/2119093778 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.550 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 -- 172.21.15.63:0/2119093778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b75a00 msgr2=0x55fb60b8dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b75a00 0x55fb60b8dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 -- 172.21.15.63:0/2119093778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 msgr2=0x55fb60b6c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b6c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 -- 172.21.15.63:0/2119093778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb60b62690 con 0x55fb60b6a820 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd543400640 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd53401c040 tx=0x7fd5340197a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94fb1bdf8fde620 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd533e00640 1 -- 172.21.15.63:0/2119093778 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53401e040 con 0x55fb60b6a820 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd533e00640 1 -- 172.21.15.63:0/2119093778 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53400faf0 con 0x55fb60b6a820 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.548+0000 7fd533e00640 1 -- 172.21.15.63:0/2119093778 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534019a70 con 0x55fb60b6a820 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 -- 172.21.15.63:0/2119093778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 msgr2=0x55fb60b814f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd53401c040 tx=0x7fd5340197a0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 -- 172.21.15.63:0/2119093778 shutdown_connections 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b75a00 0x55fb60b8dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6b140 0x55fb60b6c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 --2- 172.21.15.63:0/2119093778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b6a820 0x55fb60b814f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.549+0000 7fd54d19c740 1 -- 172.21.15.63:0/2119093778 >> 172.21.15.63:0/2119093778 conn(0x55fb60b66d30 msgr2=0x55fb60b6c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.550+0000 7fd54d19c740 1 -- 172.21.15.63:0/2119093778 shutdown_connections 2024-04-25T18:53:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.550+0000 7fd54d19c740 1 -- 172.21.15.63:0/2119093778 wait complete. 2024-04-25T18:53:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.550+0000 7fd54d19c740 1 -- start start 2024-04-25T18:53:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.550+0000 7fd54d19c740 1 Processor -- start 2024-04-25T18:53:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 0x55fb60b7dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b7b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441ae440 con 0x55fb60b6a820 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441af8b0 con 0x55fb60b75a00 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd5441aeb10 con 0x55fb60b6b140 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd54d19c740 10 client.0 _ll_register_callbacks cb 0x7fd53c0bd550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd53be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd543400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 0x55fb60b7dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd53be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:60862/0 (socket says 172.21.15.63:60862) 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd53be00640 1 -- 172.21.15.63:0/3479705731 learned_addr learned my addr 172.21.15.63:0/3479705731 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.551+0000 7fd543e00640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b7b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 -- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 msgr2=0x55fb60b7b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b7b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 -- 172.21.15.63:0/3479705731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 msgr2=0x55fb60b7dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 0x55fb60b7dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 -- 172.21.15.63:0/3479705731 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb60b7beb0 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd543e00640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b7b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd543400640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 0x55fb60b7dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd53be00640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fb60b67ec0 tx=0x7fd52c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=591f3f46406abfe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd531600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c003dc0 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd53400faf0 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.552+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fd52c01a070 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd52c02e9b0 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 10 client.26083 fetch_fsmap learned FSMap version 33 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fb60bec440 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53400faf0 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 33) ==== 8354+0+0 (secure 0 0 0) 0x7fd52c005160 con 0x55fb60b6b140 2024-04-25T18:53:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 10 client.26083 fetch_fsmap finished waiting for FSMap version 33 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 10 client.26083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 20 client.26083 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60becbd0 0x55fb60beeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 4 client.26083 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.553+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb60ade320 con 0x55fb60becbd0 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.554+0000 7fd543400640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60becbd0 0x55fb60beeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.554+0000 7fd531600640 10 client.26083 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.554+0000 7fd543400640 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60becbd0 0x55fb60beeff0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66200459677c7db9 server_cookie=37da1435bd623633 in_seq=0 out_seq=0 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd531600640 1 -- 172.21.15.63:0/3479705731 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb60ade320 con 0x55fb60becbd0 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd531600640 10 client.26083 handle_command_reply: tid=0 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 client.26083 shutdown 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 msgr2=0x55fb60b7b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fb60b67ec0 tx=0x7fd52c004730 comp rx=0 tx=0).stop 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 shutdown_connections 2024-04-25T18:53:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fb60becbd0 0x55fb60beeff0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fb60b75a00 0x55fb60b7b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fb60b6b140 0x55fb60b7b170 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 --2- 172.21.15.63:0/3479705731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fb60b6a820 0x55fb60b7dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd531600640 0 client.26083 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.608+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 >> 172.21.15.63:0/3479705731 conn(0x55fb60b66d30 msgr2=0x55fb60b6ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.609+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 shutdown_connections 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.609+0000 7fd54d19c740 1 -- 172.21.15.63:0/3479705731 wait complete. 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.609+0000 7fd54d19c740 20 client.26083 trim_cache size 0 max 16384 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:53:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:53:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:53:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:53:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 msgr2=0x7fd5280a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 0x7fd5280a6300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x55fb60b68150 tx=0x7fd52c013040 comp rx=0 tx=0).stop 2024-04-25T18:53:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 msgr2=0x7fd54419ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd53c005cc0 tx=0x7fd53c031700 comp rx=0 tx=0).stop 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 shutdown_connections 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd5280a3e40 0x7fd5280a6300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd54419a6c0 0x7fd54419ab40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd544109470 0x7fd5441a0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 --2- 172.21.15.63:0/1980872135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd5441043a0 0x7fd5441a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 >> 172.21.15.63:0/1980872135 conn(0x7fd5440ffc40 msgr2=0x7fd544101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 shutdown_connections 2024-04-25T18:53:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:53:52.610+0000 7fd54a400640 1 -- 172.21.15.63:0/1980872135 wait complete. 2024-04-25T18:53:52.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:52 smithi063 ceph-mon[28006]: pgmap v225: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:52.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:52 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:52 smithi114 ceph-mon[32988]: pgmap v225: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:52 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 -- 172.21.15.63:0/3575010428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234108a80 msgr2=0x7f9234108e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 --2- 172.21.15.63:0/3575010428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234108a80 0x7f9234108e60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9224009990 tx=0x7f9224009ca0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 -- 172.21.15.63:0/3575010428 shutdown_connections 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 --2- 172.21.15.63:0/3575010428 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234103450 0x7f923410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 --2- 172.21.15.63:0/3575010428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234108a80 0x7f9234108e60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 --2- 172.21.15.63:0/3575010428 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f9234102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 -- 172.21.15.63:0/3575010428 >> 172.21.15.63:0/3575010428 conn(0x7f9234079bf0 msgr2=0x7f923407a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 -- 172.21.15.63:0/3575010428 shutdown_connections 2024-04-25T18:53:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.941+0000 7f9238c00700 1 -- 172.21.15.63:0/3575010428 wait complete. 2024-04-25T18:53:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.942+0000 7f9238c00700 1 Processor -- start 2024-04-25T18:53:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.942+0000 7f9238c00700 1 -- start start 2024-04-25T18:53:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234103450 0x7f923419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234108a80 0x7f923419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9234112c60 con 0x7f9234103450 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9234112ae0 con 0x7f9234102a90 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9238c00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9234112de0 con 0x7f9234108a80 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9231e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9231e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53690/0 (socket says 172.21.15.63:53690) 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9231e00700 1 -- 172.21.15.63:0/546966374 learned_addr learned my addr 172.21.15.63:0/546966374 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9232800700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234108a80 0x7f923419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.943+0000 7f9231400700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234103450 0x7f923419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 -- 172.21.15.63:0/546966374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234108a80 msgr2=0x7f923419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234108a80 0x7f923419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 -- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234103450 msgr2=0x7f923419b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234103450 0x7f923419b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 -- 172.21.15.63:0/546966374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92341a03e0 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9231e00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f92240057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224027070 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92341a0670 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9224022470 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.944+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224017650 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.945+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92341a7f50 con 0x7f9234102a90 2024-04-25T18:53:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.946+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9234069730 con 0x7f9234102a90 2024-04-25T18:53:52.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.948+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f922402a070 con 0x7f9234102a90 2024-04-25T18:53:52.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.949+0000 7f922aa00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 0x7f92180a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:52.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.949+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7f92240d1550 con 0x7f9234102a90 2024-04-25T18:53:52.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.950+0000 7f9231400700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 0x7f92180a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:52.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.950+0000 7f9231400700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 0x7f92180a62f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9220009680 tx=0x7f9220009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:52.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:52 smithi070 ceph-mon[32820]: pgmap v225: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:52.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:52 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:53:52.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:52.954+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9224013070 con 0x7f9234102a90 2024-04-25T18:53:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.266+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92341a8120 con 0x7f9234102a90 2024-04-25T18:53:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.266+0000 7f922aa00700 1 -- 172.21.15.63:0/546966374 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11874 (secure 0 0 0) 0x7f922409dc40 con 0x7f9234102a90 2024-04-25T18:53:53.268 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:53:53.268 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:53:11.163286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.269+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 msgr2=0x7f92180a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.269+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 0x7f92180a62f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9220009680 tx=0x7f9220009260 comp rx=0 tx=0).stop 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.269+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 msgr2=0x7f923419b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.269+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9224000c00 tx=0x7f92240057c0 comp rx=0 tx=0).stop 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 shutdown_connections 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f92180a3e30 0x7f92180a62f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9234108a80 0x7f923419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9234102a90 0x7f923419b3d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 --2- 172.21.15.63:0/546966374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9234103450 0x7f923419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 >> 172.21.15.63:0/546966374 conn(0x7f9234079bf0 msgr2=0x7f923410df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 shutdown_connections 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:53.270+0000 7f9238c00700 1 -- 172.21.15.63:0/546966374 wait complete. 2024-04-25T18:53:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 33 2024-04-25T18:53:53.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 252481 4.36 MB/sec execute 232 sec latency 100.387 ms 2024-04-25T18:53:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:53 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/546966374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:53:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:53 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:53 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:53.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:53 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:53:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:53 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/546966374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:53:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:53 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:53 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:53.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:53 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:53:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:53 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/546966374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:53:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:53 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:53:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:53 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:53:53.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:53 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:53:54.136 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 77.4 --await-for 42.0' 2024-04-25T18:53:54.136 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 77.4 --await-for 42.0 2024-04-25T18:53:54.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 253770 4.37 MB/sec execute 233 sec latency 43.985 ms 2024-04-25T18:53:54.633 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:53:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:54 smithi114 ceph-mon[32988]: pgmap v226: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:54.844 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:54 smithi063 ceph-mon[28006]: pgmap v226: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:54.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:54 smithi070 ceph-mon[32820]: pgmap v226: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-25T18:53:55.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 255529 4.40 MB/sec execute 234 sec latency 17.791 ms 2024-04-25T18:53:55.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:55 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:55.832 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:55 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:55.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:55 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:53:56.475 INFO:tasks.workunit.client.0.smithi063.stdout: 1 256333 4.39 MB/sec execute 235 sec latency 25.391 ms 2024-04-25T18:53:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:56 smithi063 ceph-mon[28006]: pgmap v227: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 256 op/s 2024-04-25T18:53:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:56 smithi114 ceph-mon[32988]: pgmap v227: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 256 op/s 2024-04-25T18:53:56.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:56 smithi070 ceph-mon[32820]: pgmap v227: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 256 op/s 2024-04-25T18:53:57.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.259+0000 7fc623e00700 1 -- 172.21.15.63:0/336673150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c103480 msgr2=0x7fc61c10f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.259+0000 7fc623e00700 1 --2- 172.21.15.63:0/336673150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c103480 0x7fc61c10f930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b0a0 tx=0x7fc61400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 -- 172.21.15.63:0/336673150 shutdown_connections 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 --2- 172.21.15.63:0/336673150 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c103480 0x7fc61c10f930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 --2- 172.21.15.63:0/336673150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c108a60 0x7fc61c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 --2- 172.21.15.63:0/336673150 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c102ac0 0x7fc61c102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 -- 172.21.15.63:0/336673150 >> 172.21.15.63:0/336673150 conn(0x7fc61c079b60 msgr2=0x7fc61c079f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 -- 172.21.15.63:0/336673150 shutdown_connections 2024-04-25T18:53:57.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.260+0000 7fc623e00700 1 -- 172.21.15.63:0/336673150 wait complete. 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.261+0000 7fc623e00700 1 Processor -- start 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.261+0000 7fc623e00700 1 -- start start 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c103480 0x7fc61c075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c108a60 0x7fc61c0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112ca0 con 0x7fc61c103480 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112b20 con 0x7fc61c102ac0 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc623e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c112e20 con 0x7fc61c108a60 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53702/0 (socket says 172.21.15.63:53702) 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 -- 172.21.15.63:0/337793944 learned_addr learned my addr 172.21.15.63:0/337793944 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c108a60 0x7fc61c0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc620a00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c103480 0x7fc61c075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 -- 172.21.15.63:0/337793944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c108a60 msgr2=0x7fc61c0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c108a60 0x7fc61c0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 -- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c103480 msgr2=0x7fc61c075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c103480 0x7fc61c075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.262+0000 7fc621400700 1 -- 172.21.15.63:0/337793944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c1a7540 con 0x7fc61c102ac0 2024-04-25T18:53:57.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc621400700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61000e9a0 tx=0x7fc61000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc61a000700 1 -- 172.21.15.63:0/337793944 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61000cbd0 con 0x7fc61c102ac0 2024-04-25T18:53:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc61a000700 1 -- 172.21.15.63:0/337793944 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc61000cd70 con 0x7fc61c102ac0 2024-04-25T18:53:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc61a000700 1 -- 172.21.15.63:0/337793944 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc610018680 con 0x7fc61c102ac0 2024-04-25T18:53:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c1a7830 con 0x7fc61c102ac0 2024-04-25T18:53:57.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.263+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c1a7d20 con 0x7fc61c102ac0 2024-04-25T18:53:57.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.268+0000 7fc61a000700 1 -- 172.21.15.63:0/337793944 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc610010430 con 0x7fc61c102ac0 2024-04-25T18:53:57.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc62e72eb80 1 Processor -- start 2024-04-25T18:53:57.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc61a000700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 0x7fc6040a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc62e72eb80 1 -- start start 2024-04-25T18:53:57.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc61a000700 1 -- 172.21.15.63:0/337793944 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fc6100d09e0 con 0x7fc61c102ac0 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc621e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 0x7fc6040a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3163fdd0 0x563a316401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.270+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3164b560 0x563a3164d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563a31593020 con 0x563a3163fdd0 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a315398c0 con 0x563a316406f0 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a3158efe0 con 0x563a3164b560 2024-04-25T18:53:57.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc621400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3163fdd0 0x563a316401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:53706/0 (socket says 172.21.15.63:53706) 2024-04-25T18:53:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 -- 172.21.15.63:0/3551607132 learned_addr learned my addr 172.21.15.63:0/3551607132 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc621e00700 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3164b560 0x563a3164d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 -- 172.21.15.63:0/3551607132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3164b560 msgr2=0x563a3164d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3164b560 0x563a3164d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 -- 172.21.15.63:0/3551607132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3163fdd0 msgr2=0x563a316401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3163fdd0 0x563a316401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.271+0000 7fc620a00700 1 -- 172.21.15.63:0/3551607132 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a3152f810 con 0x563a316406f0 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.272+0000 7fc621e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 0x7fc6040a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b780 tx=0x7fc61400bb90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.272+0000 7fc620a00700 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc608009c80 tx=0x7fc608009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3697e1bc806b7149 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.272+0000 7fc60ea00700 1 -- 172.21.15.63:0/3551607132 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608011bf0 con 0x563a316406f0 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.272+0000 7fc60ea00700 1 -- 172.21.15.63:0/3551607132 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6080044d0 con 0x563a316406f0 2024-04-25T18:53:57.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.272+0000 7fc60ea00700 1 -- 172.21.15.63:0/3551607132 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608012450 con 0x563a316406f0 2024-04-25T18:53:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 -- 172.21.15.63:0/3551607132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 msgr2=0x563a3164b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc608009c80 tx=0x7fc608009f90 comp rx=0 tx=0).stop 2024-04-25T18:53:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 -- 172.21.15.63:0/3551607132 shutdown_connections 2024-04-25T18:53:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3164b560 0x563a3164d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3163fdd0 0x563a316401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/3551607132 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a3164b020 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 -- 172.21.15.63:0/3551607132 >> 172.21.15.63:0/3551607132 conn(0x563a3156f0e0 msgr2=0x563a3163c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 -- 172.21.15.63:0/3551607132 shutdown_connections 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.273+0000 7fc62e72eb80 1 -- 172.21.15.63:0/3551607132 wait complete. 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.274+0000 7fc62e72eb80 1 -- start start 2024-04-25T18:53:57.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.274+0000 7fc62e72eb80 1 Processor -- start 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a31648040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a31648580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563a3164dff0 con 0x563a3164b560 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563a3154c9f0 con 0x563a316406f0 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563a3164e170 con 0x563a3163fdd0 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc62e72eb80 10 client.0 _ll_register_callbacks cb 0x563a3152b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc620a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a31648040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:60904/0 (socket says 172.21.15.63:60904) 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc621400700 1 -- 172.21.15.63:0/2186614975 learned_addr learned my addr 172.21.15.63:0/2186614975 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:57.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc621e00700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a31648580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc620a00700 1 -- 172.21.15.63:0/2186614975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 msgr2=0x563a3164ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc620a00700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc620a00700 1 -- 172.21.15.63:0/2186614975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 msgr2=0x563a31648580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.275+0000 7fc620a00700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a31648580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.276+0000 7fc620a00700 1 -- 172.21.15.63:0/2186614975 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a314e0a60 con 0x563a316406f0 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.276+0000 7fc621e00700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a31648580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:53:57.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.276+0000 7fc621400700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.276+0000 7fc620a00700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a31648040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc608003aa0 tx=0x7fc6080119f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe34df25897cc9e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.276+0000 7fc603400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608027020 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6080044d0 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fc608005050 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc60802f050 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc62e72eb80 10 client.26078 fetch_fsmap learned FSMap version 34 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.277+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a31529c80 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6080044d0 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fc608003b00 con 0x563a316406f0 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 10 client.26078 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:53:57.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 10 client.26078 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 20 client.26078 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a3165abd0 0x563a3165cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 4 client.26078 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a315a0810 con 0x563a3165abd0 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.278+0000 7fc621400700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a3165abd0 0x563a3165cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.279+0000 7fc621400700 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a3165abd0 0x563a3165cfc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c383705ec6822fda server_cookie=9af4caf88ea23fd2 in_seq=0 out_seq=0 2024-04-25T18:53:57.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.279+0000 7fc603400700 10 client.26078 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.279+0000 7fc603400700 1 -- 172.21.15.63:0/2186614975 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563a315a0810 con 0x563a3165abd0 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.279+0000 7fc603400700 10 client.26078 handle_command_reply: tid=0 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 client.26078 shutdown 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 msgr2=0x563a31648040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a31648040 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc608003aa0 tx=0x7fc6080119f0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 shutdown_connections 2024-04-25T18:53:57.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a3165abd0 0x563a3165cfc0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a31648580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/2186614975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a316406f0 0x563a31648040 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc603400700 0 client.26078 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 >> 172.21.15.63:0/2186614975 conn(0x563a3156f0e0 msgr2=0x563a3163bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 shutdown_connections 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 1 -- 172.21.15.63:0/2186614975 wait complete. 2024-04-25T18:53:57.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.280+0000 7fc62e72eb80 20 client.26078 trim_cache size 0 max 16384 2024-04-25T18:53:57.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 Processor -- start 2024-04-25T18:53:57.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 -- start start 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a3164b560 0x563a31656ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a31662620 0x563a31662a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1b1f10 con 0x563a31662620 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.302+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1af740 con 0x563a3164b560 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ad5b0 con 0x563a3163fdd0 2024-04-25T18:53:57.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a31662620 0x563a31662a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43664/0 (socket says 172.21.15.63:43664) 2024-04-25T18:53:57.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 -- 172.21.15.63:0/120448371 learned_addr learned my addr 172.21.15.63:0/120448371 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:57.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc620a00700 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a3164b560 0x563a31656ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 -- 172.21.15.63:0/120448371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a3164b560 msgr2=0x563a31656ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a3164b560 0x563a31656ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 -- 172.21.15.63:0/120448371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a31662620 msgr2=0x563a31662a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a31662620 0x563a31662a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.303+0000 7fc621400700 1 -- 172.21.15.63:0/120448371 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a31640400 con 0x563a3163fdd0 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.304+0000 7fc621400700 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc61009c870 tx=0x7fc6100d0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b88c1a5f0a15079 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.304+0000 7fc60e000700 1 -- 172.21.15.63:0/120448371 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61009c8e0 con 0x563a3163fdd0 2024-04-25T18:53:57.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.304+0000 7fc60e000700 1 -- 172.21.15.63:0/120448371 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6100a0c90 con 0x563a3163fdd0 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.304+0000 7fc60e000700 1 -- 172.21.15.63:0/120448371 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100a5320 con 0x563a3163fdd0 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- 172.21.15.63:0/120448371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 msgr2=0x563a3164a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc61009c870 tx=0x7fc6100d0040 comp rx=0 tx=0).stop 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- 172.21.15.63:0/120448371 shutdown_connections 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164a250 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a31662620 0x563a31662a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/120448371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a3164b560 0x563a31656ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- 172.21.15.63:0/120448371 >> 172.21.15.63:0/120448371 conn(0x563a3156f0e0 msgr2=0x563a31647090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- 172.21.15.63:0/120448371 shutdown_connections 2024-04-25T18:53:57.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- 172.21.15.63:0/120448371 wait complete. 2024-04-25T18:53:57.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.305+0000 7fc62e72eb80 1 -- start start 2024-04-25T18:53:57.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 Processor -- start 2024-04-25T18:53:57.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a316504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a31662620 0x563a31650a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1adc00 con 0x563a3164b560 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1af070 con 0x563a31662620 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c1ae2d0 con 0x563a3163fdd0 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc62e72eb80 10 client.0 _ll_register_callbacks cb 0x563a3152b8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc621400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43676/0 (socket says 172.21.15.63:43676) 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc621400700 1 -- 172.21.15.63:0/439439270 learned_addr learned my addr 172.21.15.63:0/439439270 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.306+0000 7fc621e00700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a31662620 0x563a31650a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc620a00700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a316504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 -- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a31662620 msgr2=0x563a31650a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a31662620 0x563a31650a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 -- 172.21.15.63:0/439439270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 msgr2=0x563a316504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:53:57.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a316504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 -- 172.21.15.63:0/439439270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a316c1010 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc621400700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61009c5e0 tx=0x7fc6100a55d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5bb2c744fdd3ec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc603400700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100a50c0 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 8587+0+0 (secure 0 0 0) 0x7fc6100a5260 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6100a0c90 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.307+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fc610010720 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 10 client.26107 fetch_fsmap learned FSMap version 34 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563a316c22b0 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6100a0c90 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fc61009b7a0 con 0x563a3163fdd0 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 10 client.26107 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 10 client.26107 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 20 client.26107 populate_metadata read hostname 'smithi063' 2024-04-25T18:53:57.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a316c27d0 0x563a316c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:53:57.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 4 client.26107 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.4, "await_for": 42.0}] 2024-04-25T18:53:57.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 77.4, "await_for": 42.0}) -- 0x563a3159a8b0 con 0x563a316c27d0 2024-04-25T18:53:57.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.308+0000 7fc620a00700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a316c27d0 0x563a316c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:53:57.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.309+0000 7fc620a00700 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a316c27d0 0x563a316c4bf0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d40792c12a4c98c4 server_cookie=611b53108d69201f in_seq=0 out_seq=0 2024-04-25T18:53:57.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.309+0000 7fc603400700 10 client.26107 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:57.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc603400700 1 -- 172.21.15.63:0/439439270 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x563a3159a8b0 con 0x563a316c27d0 2024-04-25T18:53:57.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc603400700 10 client.26107 handle_command_reply: tid=0 2024-04-25T18:53:57.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 client.26107 shutdown 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 msgr2=0x563a3164ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61009c5e0 tx=0x7fc6100a55d0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 shutdown_connections 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563a316c27d0 0x563a316c4bf0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563a3163fdd0 0x563a3164ffa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc603400700 0 client.26107 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.375+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563a31662620 0x563a31650a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.376+0000 7fc62e72eb80 1 --2- 172.21.15.63:0/439439270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563a3164b560 0x563a316504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.376+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 >> 172.21.15.63:0/439439270 conn(0x563a3156f0e0 msgr2=0x563a31646800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.376+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 shutdown_connections 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.376+0000 7fc62e72eb80 1 -- 172.21.15.63:0/439439270 wait complete. 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.376+0000 7fc62e72eb80 20 client.26107 trim_cache size 0 max 16384 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 34, 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 22, 2024-04-25T18:53:57.379 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "c02b8367": { 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 22, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 77.4, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:53:57.380 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:53:57.381 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.377+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 msgr2=0x7fc6040a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.377+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 0x7fc6040a6390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc61400b780 tx=0x7fc61400bb90 comp rx=0 tx=0).stop 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.377+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 msgr2=0x7fc61c074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.377+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc61000e9a0 tx=0x7fc61000ecb0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 shutdown_connections 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc6040a3ed0 0x7fc6040a6390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc61c108a60 0x7fc61c0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc61c102ac0 0x7fc61c074d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 --2- 172.21.15.63:0/337793944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc61c103480 0x7fc61c075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:53:57.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.378+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 >> 172.21.15.63:0/337793944 conn(0x7fc61c079b60 msgr2=0x7fc61c1009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:53:57.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.379+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 shutdown_connections 2024-04-25T18:53:57.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:53:57.379+0000 7fc623e00700 1 -- 172.21.15.63:0/337793944 wait complete. 2024-04-25T18:53:57.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.40 MB/sec execute 236 sec latency 101.440 ms 2024-04-25T18:53:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:53:57 smithi063 ceph-mon[28006]: pgmap v228: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-25T18:53:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:53:57 smithi070 ceph-mon[32820]: pgmap v228: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-25T18:53:58.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:53:57 smithi114 ceph-mon[32988]: pgmap v228: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-25T18:53:58.337 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c02b8367', quiesce took 8.4 seconds. Will release after: 17.4 2024-04-25T18:53:58.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.38 MB/sec execute 237 sec latency 1101.568 ms 2024-04-25T18:53:59.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.36 MB/sec execute 238 sec latency 2101.666 ms 2024-04-25T18:54:00.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:00 smithi070 ceph-mon[32820]: pgmap v229: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-04-25T18:54:00.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.34 MB/sec execute 239 sec latency 3101.827 ms 2024-04-25T18:54:00.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:00 smithi114 ceph-mon[32988]: pgmap v229: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-04-25T18:54:00.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:00 smithi063 ceph-mon[28006]: pgmap v229: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 200 op/s 2024-04-25T18:54:01.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.32 MB/sec execute 240 sec latency 4101.917 ms 2024-04-25T18:54:02.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:02 smithi070 ceph-mon[32820]: pgmap v230: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:02.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.31 MB/sec execute 241 sec latency 5102.059 ms 2024-04-25T18:54:02.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:02 smithi114 ceph-mon[32988]: pgmap v230: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:02.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:02 smithi063 ceph-mon[28006]: pgmap v230: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:03.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.29 MB/sec execute 242 sec latency 6102.211 ms 2024-04-25T18:54:04.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:04 smithi070 ceph-mon[32820]: pgmap v231: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:04.476 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.27 MB/sec execute 243 sec latency 7102.291 ms 2024-04-25T18:54:04.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:04 smithi114 ceph-mon[32988]: pgmap v231: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:04.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:04 smithi063 ceph-mon[28006]: pgmap v231: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1.4 MiB/s wr, 144 op/s 2024-04-25T18:54:05.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:05 smithi070 ceph-mon[32820]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T18:54:05.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.25 MB/sec execute 244 sec latency 8102.459 ms 2024-04-25T18:54:05.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:05 smithi114 ceph-mon[32988]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T18:54:05.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:05 smithi063 ceph-mon[28006]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T18:54:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 -- 172.21.15.63:0/1091497521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074c20 msgr2=0x7f78ac10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 --2- 172.21.15.63:0/1091497521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074c20 0x7f78ac10e0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f789c009900 tx=0x7f789c02a480 comp rx=0 tx=0).stop 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 -- 172.21.15.63:0/1091497521 shutdown_connections 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 --2- 172.21.15.63:0/1091497521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 0x7f78ac110a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 --2- 172.21.15.63:0/1091497521 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074c20 0x7f78ac10e0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 --2- 172.21.15.63:0/1091497521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074300 0x7f78ac0746e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 -- 172.21.15.63:0/1091497521 >> 172.21.15.63:0/1091497521 conn(0x7f78ac0fd890 msgr2=0x7f78ac0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 -- 172.21.15.63:0/1091497521 shutdown_connections 2024-04-25T18:54:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.759+0000 7f78b3200640 1 -- 172.21.15.63:0/1091497521 wait complete. 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 Processor -- start 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 -- start start 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074c20 0x7f78ac1a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 0x7f78ac19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac114230 con 0x7f78ac10e620 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1140b0 con 0x7f78ac074c20 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.760+0000 7f78b3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1143b0 con 0x7f78ac074300 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 0x7f78ac19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43696/0 (socket says 172.21.15.63:43696) 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 -- 172.21.15.63:0/2641055171 learned_addr learned my addr 172.21.15.63:0/2641055171 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1400640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074c20 0x7f78ac1a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 -- 172.21.15.63:0/2641055171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074c20 msgr2=0x7f78ac1a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074c20 0x7f78ac1a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 -- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 msgr2=0x7f78ac19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 0x7f78ac19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 -- 172.21.15.63:0/2641055171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78ac19af10 con 0x7f78ac074300 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.761+0000 7f78b1e00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f78a000e9f0 tx=0x7f78a000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.762+0000 7f78a6a00640 1 -- 172.21.15.63:0/2641055171 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a000ccc0 con 0x7f78ac074300 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.762+0000 7f78a6a00640 1 -- 172.21.15.63:0/2641055171 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a000ce60 con 0x7f78ac074300 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.762+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78ac19b200 con 0x7f78ac074300 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.762+0000 7f78a6a00640 1 -- 172.21.15.63:0/2641055171 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a0010430 con 0x7f78ac074300 2024-04-25T18:54:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.762+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78ac1a7510 con 0x7f78ac074300 2024-04-25T18:54:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.765+0000 7f78b77ba740 1 Processor -- start 2024-04-25T18:54:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.765+0000 7f78b77ba740 1 -- start start 2024-04-25T18:54:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.765+0000 7f78a6a00640 1 -- 172.21.15.63:0/2641055171 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f78a00046c0 con 0x7f78ac074300 2024-04-25T18:54:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.765+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 0x5620f3993f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b77ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5620f38ec900 con 0x5620f3991b90 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b77ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5620f3892ef0 con 0x5620f39872d0 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b77ba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5620f38e88c0 con 0x5620f39869b0 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 0x5620f3993f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43698/0 (socket says 172.21.15.63:43698) 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58794/0 (socket says 172.21.15.63:58794) 2024-04-25T18:54:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.766+0000 7f78b2800640 1 -- 172.21.15.63:0/3306140307 learned_addr learned my addr 172.21.15.63:0/3306140307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 -- 172.21.15.63:0/3306140307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 msgr2=0x5620f3991650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 -- 172.21.15.63:0/3306140307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 msgr2=0x5620f3993f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 0x5620f3993f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 -- 172.21.15.63:0/3306140307 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620f35ff210 con 0x5620f39869b0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b1e00640 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b1400640 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 0x5620f3993f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b2800640 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009f60 tx=0x7f78a8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6e58b73d4b97454 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78a6a00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 0x7f78940a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f789b000640 1 -- 172.21.15.63:0/3306140307 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a801e070 con 0x5620f39869b0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f789b000640 1 -- 172.21.15.63:0/3306140307 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a8003c30 con 0x5620f39869b0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f789b000640 1 -- 172.21.15.63:0/3306140307 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a800f840 con 0x5620f39869b0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78a6a00640 1 -- 172.21.15.63:0/2641055171 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8631+0+0 (secure 0 0 0) 0x7f78a00d0ba0 con 0x7f78ac074300 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.767+0000 7f78b1e00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 0x7f78940a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 -- 172.21.15.63:0/3306140307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 msgr2=0x5620f3986d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009f60 tx=0x7f78a8009f90 comp rx=0 tx=0).stop 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 -- 172.21.15.63:0/3306140307 shutdown_connections 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b1e00640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 0x7f78940a6250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ce330 tx=0x7f78a00d46e0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3991650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f3991b90 0x5620f3993f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3306140307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39869b0 0x5620f3986d90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 -- 172.21.15.63:0/3306140307 >> 172.21.15.63:0/3306140307 conn(0x5620f3982ec0 msgr2=0x5620f39832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 -- 172.21.15.63:0/3306140307 shutdown_connections 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.768+0000 7f78b77ba740 1 -- 172.21.15.63:0/3306140307 wait complete. 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 -- start start 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 Processor -- start 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f398eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 0x5620f398f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f3991b90 0x5620f398f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5620f39947c0 con 0x5620f39869b0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5620f38a6310 con 0x5620f3991b90 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5620f3994940 con 0x5620f39872d0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.769+0000 7f78b77ba740 10 client.0 _ll_register_callbacks cb 0x5620f3874cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f398eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f398eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51738/0 (socket says 172.21.15.63:51738) 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b2800640 1 -- 172.21.15.63:0/3968288842 learned_addr learned my addr 172.21.15.63:0/3968288842 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 0x5620f398f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b2800640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f3991b90 0x5620f398f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 -- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f3991b90 msgr2=0x5620f398f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f3991b90 0x5620f398f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 -- 172.21.15.63:0/3968288842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 msgr2=0x5620f398eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f398eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.770+0000 7f78b1400640 1 -- 172.21.15.63:0/3968288842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620f36cb620 con 0x5620f39872d0 2024-04-25T18:54:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f78b1400640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 0x5620f398f0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f789c02aa70 tx=0x7f789c004440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8f583c91973761 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f788f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c035040 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8631+0+0 (secure 0 0 0) 0x7f789c005510 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78a8003c30 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f789c004770 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f78b77ba740 10 client.26119 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.771+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5620f39a0a80 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78a8003c30 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f789c003a10 con 0x5620f39872d0 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 10 client.26119 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 10 client.26119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 20 client.26119 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:05.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f39a1f40 0x5620f39a4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 4 client.26119 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5620f39008b0 con 0x5620f39a1f40 2024-04-25T18:54:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.772+0000 7f78b2800640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f39a1f40 0x5620f39a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.773+0000 7f78b2800640 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f39a1f40 0x5620f39a4330 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71d43d6902c8385b server_cookie=37ef059a7af66ad5 in_seq=0 out_seq=0 2024-04-25T18:54:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.773+0000 7f788f400640 10 client.26119 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f788f400640 1 -- 172.21.15.63:0/3968288842 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5620f39008b0 con 0x5620f39a1f40 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f788f400640 10 client.26119 handle_command_reply: tid=0 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 client.26119 shutdown 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 msgr2=0x5620f398f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 0x5620f398f0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f789c02aa70 tx=0x7f789c004440 comp rx=0 tx=0).stop 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 shutdown_connections 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f39a1f40 0x5620f39a4330 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f3991b90 0x5620f398f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f39872d0 0x5620f398f0c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3968288842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f398eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f788f400640 0 client.26119 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 >> 172.21.15.63:0/3968288842 conn(0x5620f3982ec0 msgr2=0x5620f3988bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 shutdown_connections 2024-04-25T18:54:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 1 -- 172.21.15.63:0/3968288842 wait complete. 2024-04-25T18:54:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.774+0000 7f78b77ba740 20 client.26119 trim_cache size 0 max 16384 2024-04-25T18:54:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 Processor -- start 2024-04-25T18:54:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 -- start start 2024-04-25T18:54:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 0x5620f399d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.794 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39aa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1b2740 con 0x5620f39872d0 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1affb0 con 0x5620f39869b0 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b77ba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1ade20 con 0x5620f3991b90 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51752/0 (socket says 172.21.15.63:51752) 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b1400640 1 -- 172.21.15.63:0/3770013498 learned_addr learned my addr 172.21.15.63:0/3770013498 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.791+0000 7f78b2800640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 0x5620f399d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b2800640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 -- 172.21.15.63:0/3770013498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 msgr2=0x5620f39aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 -- 172.21.15.63:0/3770013498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 msgr2=0x5620f399d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 0x5620f399d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 -- 172.21.15.63:0/3770013498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620f39827a0 con 0x5620f39872d0 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b2800640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 0x5620f399d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f78b1400640 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f789c000c00 tx=0x7f789c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719cdadb9b5752bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f789a600640 1 -- 172.21.15.63:0/3770013498 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c035020 con 0x5620f39872d0 2024-04-25T18:54:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.792+0000 7f789a600640 1 -- 172.21.15.63:0/3770013498 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c041070 con 0x5620f39872d0 2024-04-25T18:54:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f789a600640 1 -- 172.21.15.63:0/3770013498 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0035f0 con 0x5620f39872d0 2024-04-25T18:54:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 -- 172.21.15.63:0/3770013498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 msgr2=0x5620f39a9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f789c000c00 tx=0x7f789c031630 comp rx=0 tx=0).stop 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 -- 172.21.15.63:0/3770013498 shutdown_connections 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39872d0 0x5620f39a9c00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 --2- 172.21.15.63:0/3770013498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39869b0 0x5620f399d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 -- 172.21.15.63:0/3770013498 >> 172.21.15.63:0/3770013498 conn(0x5620f3982ec0 msgr2=0x5620f3989e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 -- 172.21.15.63:0/3770013498 shutdown_connections 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.793+0000 7f78b77ba740 1 -- 172.21.15.63:0/3770013498 wait complete. 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 -- start start 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 Processor -- start 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3997290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1ae470 con 0x5620f39869b0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1af8e0 con 0x5620f39872d0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.794+0000 7f78b77ba740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f78ac1aeb40 con 0x5620f3991b90 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b77ba740 10 client.0 _ll_register_callbacks cb 0x7f789c0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51764/0 (socket says 172.21.15.63:51764) 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 -- 172.21.15.63:0/649146345 learned_addr learned my addr 172.21.15.63:0/649146345 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b1400640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3997290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 -- 172.21.15.63:0/649146345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 msgr2=0x5620f39977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 -- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 msgr2=0x5620f3997290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3997290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 -- 172.21.15.63:0/649146345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5620f3998000 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f78b2800640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f78a8011d60 tx=0x7f78a8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758f0cea2ef77c1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.795+0000 7f788f400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a8035020 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c041070 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 8631+0+0 (secure 0 0 0) 0x7f78a802e2d0 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f78a803d050 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f78b77ba740 10 client.15921 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5620f3a08460 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f789c041070 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.796+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f78a8004570 con 0x5620f39869b0 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 10 client.15921 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 20 client.15921 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f3a089f0 0x5620f3a0ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 4 client.15921 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5620f38fa320 con 0x5620f3a089f0 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.797+0000 7f78b1400640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f3a089f0 0x5620f3a0ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.798+0000 7f78b1400640 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f3a089f0 0x5620f3a0ade0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8d59fc0beac46a server_cookie=c2f3361c4f01508d in_seq=0 out_seq=0 2024-04-25T18:54:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.798+0000 7f788f400640 10 client.15921 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f788f400640 1 -- 172.21.15.63:0/649146345 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5620f38fa320 con 0x5620f3a089f0 2024-04-25T18:54:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f788f400640 10 client.15921 handle_command_reply: tid=0 2024-04-25T18:54:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 client.15921 shutdown 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 msgr2=0x5620f3999f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f78a8011d60 tx=0x7f78a8009f90 comp rx=0 tx=0).stop 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 shutdown_connections 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5620f3a089f0 0x5620f3a0ade0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5620f39872d0 0x5620f3997290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5620f3991b90 0x5620f39977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 --2- 172.21.15.63:0/649146345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620f39869b0 0x5620f3999f00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f788f400640 0 client.15921 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 >> 172.21.15.63:0/649146345 conn(0x5620f3982ec0 msgr2=0x5620f3981790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 shutdown_connections 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 1 -- 172.21.15.63:0/649146345 wait complete. 2024-04-25T18:54:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.834+0000 7f78b77ba740 20 client.15921 trim_cache size 0 max 16384 2024-04-25T18:54:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:54:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:54:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:54:05.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 msgr2=0x7f78940a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 0x7f78940a6250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ce330 tx=0x7f78a00d46e0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 msgr2=0x7f78ac1a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f78a000e9f0 tx=0x7f78a000edf0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 shutdown_connections 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f78940a3d90 0x7f78940a6250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78ac074c20 0x7f78ac1a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78ac10e620 0x7f78ac19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 --2- 172.21.15.63:0/2641055171 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f78ac074300 0x7f78ac1a0590 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 >> 172.21.15.63:0/2641055171 conn(0x7f78ac0fd890 msgr2=0x7f78ac0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.835+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 shutdown_connections 2024-04-25T18:54:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:05.836+0000 7f78b3200640 1 -- 172.21.15.63:0/2641055171 wait complete. 2024-04-25T18:54:06.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:06 smithi070 ceph-mon[32820]: pgmap v233: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:06.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.23 MB/sec execute 245 sec latency 9102.562 ms 2024-04-25T18:54:06.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:06 smithi114 ceph-mon[32988]: pgmap v233: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:06.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:06 smithi063 ceph-mon[28006]: pgmap v233: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:07.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.22 MB/sec execute 246 sec latency 10102.665 ms 2024-04-25T18:54:08.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:08 smithi070 ceph-mon[32820]: pgmap v234: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:08.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.20 MB/sec execute 247 sec latency 11102.755 ms 2024-04-25T18:54:08.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:08 smithi114 ceph-mon[32988]: pgmap v234: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:08.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:08 smithi063 ceph-mon[28006]: pgmap v234: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-25T18:54:09.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.18 MB/sec execute 248 sec latency 12102.848 ms 2024-04-25T18:54:10.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.17 MB/sec execute 249 sec latency 13103.011 ms 2024-04-25T18:54:10.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:10 smithi114 ceph-mon[32988]: pgmap v235: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:10 smithi063 ceph-mon[28006]: pgmap v235: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:10 smithi070 ceph-mon[32820]: pgmap v235: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:11.477 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.15 MB/sec execute 250 sec latency 14103.168 ms 2024-04-25T18:54:12.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.13 MB/sec execute 251 sec latency 15103.330 ms 2024-04-25T18:54:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:12 smithi114 ceph-mon[32988]: pgmap v236: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:12.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:12 smithi063 ceph-mon[28006]: pgmap v236: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:12.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:12 smithi070 ceph-mon[32820]: pgmap v236: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:13.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.12 MB/sec execute 252 sec latency 16103.484 ms 2024-04-25T18:54:14.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.10 MB/sec execute 253 sec latency 17103.580 ms 2024-04-25T18:54:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:14 smithi114 ceph-mon[32988]: pgmap v237: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:14 smithi114 ceph-mon[32988]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T18:54:14.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:14 smithi063 ceph-mon[28006]: pgmap v237: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:14.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:14 smithi063 ceph-mon[28006]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T18:54:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:14 smithi070 ceph-mon[32820]: pgmap v237: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.1 MiB/s wr, 85 op/s 2024-04-25T18:54:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:14 smithi070 ceph-mon[32820]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T18:54:15.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.08 MB/sec execute 254 sec latency 18103.737 ms 2024-04-25T18:54:15.738 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:54:16.204 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:54:16.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.07 MB/sec execute 255 sec latency 19103.823 ms 2024-04-25T18:54:16.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:16 smithi063 ceph-mon[28006]: pgmap v239: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:16 smithi114 ceph-mon[32988]: pgmap v239: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:16 smithi070 ceph-mon[32820]: pgmap v239: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:17.478 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.05 MB/sec execute 256 sec latency 20103.959 ms 2024-04-25T18:54:18.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:18 smithi063 ceph-mon[28006]: pgmap v240: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:18.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.04 MB/sec execute 257 sec latency 21104.088 ms 2024-04-25T18:54:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:18 smithi114 ceph-mon[32988]: pgmap v240: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:18 smithi070 ceph-mon[32820]: pgmap v240: 97 pgs: 97 active+clean; 84 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 964 KiB/s wr, 63 op/s 2024-04-25T18:54:18.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 -- 172.21.15.63:0/2535781227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7010f440 msgr2=0x7f8a70111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 --2- 172.21.15.63:0/2535781227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7010f440 0x7f8a70111830 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009f90 tx=0x7f8a6800b330 comp rx=0 tx=0).stop 2024-04-25T18:54:18.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 -- 172.21.15.63:0/2535781227 shutdown_connections 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 --2- 172.21.15.63:0/2535781227 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a7010f440 0x7f8a70111830 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 --2- 172.21.15.63:0/2535781227 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70074c70 0x7f8a70075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 --2- 172.21.15.63:0/2535781227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a70075590 0x7f8a7010ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 -- 172.21.15.63:0/2535781227 >> 172.21.15.63:0/2535781227 conn(0x7f8a70079ba0 msgr2=0x7f8a70079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 -- 172.21.15.63:0/2535781227 shutdown_connections 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.799+0000 7f8a78600700 1 -- 172.21.15.63:0/2535781227 wait complete. 2024-04-25T18:54:18.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.800+0000 7f8a78600700 1 Processor -- start 2024-04-25T18:54:18.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 -- start start 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70075590 0x7f8a701030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a7010f440 0x7f8a701035e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70114e20 con 0x7f8a7010f440 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70114ca0 con 0x7f8a70075590 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a78600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70114fa0 con 0x7f8a70074c70 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a75c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a75c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51100/0 (socket says 172.21.15.63:51100) 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.801+0000 7f8a75c00700 1 -- 172.21.15.63:0/1667812526 learned_addr learned my addr 172.21.15.63:0/1667812526 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:18.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a76600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a7010f440 0x7f8a701035e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75200700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70075590 0x7f8a701030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 -- 172.21.15.63:0/1667812526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70075590 msgr2=0x7f8a701030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70075590 0x7f8a701030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 -- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a7010f440 msgr2=0x7f8a701035e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a7010f440 0x7f8a701035e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 -- 172.21.15.63:0/1667812526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a701ab9c0 con 0x7f8a70074c70 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.802+0000 7f8a75c00700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9d0 tx=0x7f8a6400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.803+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400cbd0 con 0x7f8a70074c70 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.803+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a6400cd70 con 0x7f8a70074c70 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.803+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a701abcb0 con 0x7f8a70074c70 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.803+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64018710 con 0x7f8a70074c70 2024-04-25T18:54:18.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.803+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a701ac1e0 con 0x7f8a70074c70 2024-04-25T18:54:18.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.804+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a70076360 con 0x7f8a70074c70 2024-04-25T18:54:18.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.806+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8a64010430 con 0x7f8a70074c70 2024-04-25T18:54:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.807+0000 7f8a6e000700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 0x7f8a580a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.808+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f8a640d0a40 con 0x7f8a70074c70 2024-04-25T18:54:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.808+0000 7f8a75200700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 0x7f8a580a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.808+0000 7f8a75200700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 0x7f8a580a6320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a701042d0 tx=0x7f8a5c009310 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:18.810+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a6409d040 con 0x7f8a70074c70 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 -- 172.21.15.63:0/1350552302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9430101fe0 msgr2=0x7f94301023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 --2- 172.21.15.63:0/1350552302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9430101fe0 0x7f94301023c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9420009900 tx=0x7f942002a3f0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 -- 172.21.15.63:0/1350552302 shutdown_connections 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 --2- 172.21.15.63:0/1350552302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f94301070b0 0x7f943010ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 --2- 172.21.15.63:0/1350552302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9430102990 0x7f94301069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 --2- 172.21.15.63:0/1350552302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9430101fe0 0x7f94301023c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.971+0000 7f9435c00640 1 -- 172.21.15.63:0/1350552302 >> 172.21.15.63:0/1350552302 conn(0x7f94300fd8a0 msgr2=0x7f94300ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:18.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.972+0000 7f9435c00640 1 -- 172.21.15.63:0/1350552302 shutdown_connections 2024-04-25T18:54:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.972+0000 7f9435c00640 1 -- 172.21.15.63:0/1350552302 wait complete. 2024-04-25T18:54:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 Processor -- start 2024-04-25T18:54:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 -- start start 2024-04-25T18:54:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94301070b0 0x7f94301a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f943019a720 0x7f943019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9430112040 con 0x7f94301070b0 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9430111ec0 con 0x7f943019a720 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94301121c0 con 0x7f9430101fe0 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f942fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f9435200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f943019a720 0x7f943019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f942f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94301070b0 0x7f94301a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f942fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51108/0 (socket says 172.21.15.63:51108) 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.973+0000 7f942fe00640 1 -- 172.21.15.63:0/3780262473 learned_addr learned my addr 172.21.15.63:0/3780262473 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 -- 172.21.15.63:0/3780262473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f943019a720 msgr2=0x7f943019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f943019a720 0x7f943019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 -- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94301070b0 msgr2=0x7f94301a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94301070b0 0x7f94301a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 -- 172.21.15.63:0/3780262473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943019b320 con 0x7f9430101fe0 2024-04-25T18:54:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.974+0000 7f942fe00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9420005ab0 tx=0x7f9420003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.975+0000 7f942cc00640 1 -- 172.21.15.63:0/3780262473 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942003f070 con 0x7f9430101fe0 2024-04-25T18:54:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.975+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943019b5b0 con 0x7f9430101fe0 2024-04-25T18:54:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.975+0000 7f942cc00640 1 -- 172.21.15.63:0/3780262473 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9420031b60 con 0x7f9430101fe0 2024-04-25T18:54:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.975+0000 7f942cc00640 1 -- 172.21.15.63:0/3780262473 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420035030 con 0x7f9430101fe0 2024-04-25T18:54:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.975+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94301a7740 con 0x7f9430101fe0 2024-04-25T18:54:18.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.978+0000 7f943a00b740 1 Processor -- start 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.978+0000 7f943a00b740 1 -- start start 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.978+0000 7f942cc00640 1 -- 172.21.15.63:0/3780262473 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9420042070 con 0x7f9430101fe0 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.978+0000 7f943a00b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe35d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.978+0000 7f943a00b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f943a00b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe42f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f943a00b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e50fd9b900 con 0x55e50fe40b90 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f943a00b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e50fd41ef0 con 0x55e50fe359b0 2024-04-25T18:54:18.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f943a00b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e50fd978c0 con 0x55e50fe362d0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f9435200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe35d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51116/0 (socket says 172.21.15.63:51116) 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 -- 172.21.15.63:0/1671199253 learned_addr learned my addr 172.21.15.63:0/1671199253 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942f400640 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe42f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 -- 172.21.15.63:0/1671199253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 msgr2=0x55e50fe35d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe35d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 -- 172.21.15.63:0/1671199253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 msgr2=0x55e50fe42f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe42f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.979+0000 7f942fe00640 1 -- 172.21.15.63:0/1671199253 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e50faae210 con 0x55e50fe362d0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.980+0000 7f942fe00640 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f94200ee1f0 tx=0x7f94200f4e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42106e238154f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.980+0000 7f9425600640 1 -- 172.21.15.63:0/1671199253 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420100070 con 0x55e50fe362d0 2024-04-25T18:54:18.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.980+0000 7f9425600640 1 -- 172.21.15.63:0/1671199253 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94200f9b90 con 0x55e50fe362d0 2024-04-25T18:54:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.980+0000 7f9425600640 1 -- 172.21.15.63:0/1671199253 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94200fb630 con 0x55e50fe362d0 2024-04-25T18:54:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.980+0000 7f942cc00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 0x7f94140a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f942cc00640 1 -- 172.21.15.63:0/3780262473 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f94200f0c00 con 0x7f9430101fe0 2024-04-25T18:54:18.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f9435200640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 0x7f94140a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 -- 172.21.15.63:0/1671199253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 msgr2=0x55e50fe40650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f94200ee1f0 tx=0x7f94200f4e30 comp rx=0 tx=0).stop 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 -- 172.21.15.63:0/1671199253 shutdown_connections 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe42f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe40650 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 --2- 172.21.15.63:0/1671199253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe35d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.981+0000 7f943a00b740 1 -- 172.21.15.63:0/1671199253 >> 172.21.15.63:0/1671199253 conn(0x55e50fe31ec0 msgr2=0x55e50fe322a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.982+0000 7f943a00b740 1 -- 172.21.15.63:0/1671199253 shutdown_connections 2024-04-25T18:54:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.982+0000 7f9435200640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 0x7f94140a61d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9428005fd0 tx=0x7f9428009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.982+0000 7f943a00b740 1 -- 172.21.15.63:0/1671199253 wait complete. 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.982+0000 7f943a00b740 1 -- start start 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.982+0000 7f943a00b740 1 Processor -- start 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe3e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe3e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e50fe43580 con 0x55e50fe40b90 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e50fd55310 con 0x55e50fe359b0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e50fe43700 con 0x55e50fe362d0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f943a00b740 10 client.0 _ll_register_callbacks cb 0x55e50fd23cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f9435200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe3e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60640/0 (socket says 172.21.15.63:60640) 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 -- 172.21.15.63:0/3133416226 learned_addr learned my addr 172.21.15.63:0/3133416226 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942f400640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe3e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 -- 172.21.15.63:0/3133416226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 msgr2=0x55e50fe3e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe3e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 -- 172.21.15.63:0/3133416226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 msgr2=0x55e50fe3e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe3e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.983+0000 7f942fe00640 1 -- 172.21.15.63:0/3133416226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e50fb7a620 con 0x55e50fe359b0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f942f400640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe3e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f942fe00640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f94200c5240 tx=0x7f94200045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438e1d7cc02c8761 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f941a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420049070 con 0x55e50fe359b0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f9420031d10 con 0x55e50fe359b0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.984+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94200f9b90 con 0x55e50fe359b0 2024-04-25T18:54:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9420100030 con 0x55e50fe359b0 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 10 client.26120 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e50fe43580 con 0x55e50fe359b0 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94200f9b90 con 0x55e50fe359b0 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f94200fa040 con 0x55e50fe359b0 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 10 client.26120 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 10 client.26120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 20 client.26120 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50fe50930 0x55e50fe52d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 4 client.26120 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.985+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e50fdaf8b0 con 0x55e50fe50930 2024-04-25T18:54:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.986+0000 7f942f400640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50fe50930 0x55e50fe52d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:18.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.986+0000 7f942f400640 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50fe50930 0x55e50fe52d20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9cfa333168ba53d server_cookie=d6c358b40adea99 in_seq=0 out_seq=0 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.986+0000 7f941a000640 10 client.26120 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f941a000640 1 -- 172.21.15.63:0/3133416226 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e50fdaf8b0 con 0x55e50fe50930 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f941a000640 10 client.26120 handle_command_reply: tid=0 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 client.26120 shutdown 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 msgr2=0x55e50fe3db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f94200c5240 tx=0x7f94200045e0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 shutdown_connections 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50fe50930 0x55e50fe52d20 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe40b90 0x55e50fe3e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe3e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 --2- 172.21.15.63:0/3133416226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe3db90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.987+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 >> 172.21.15.63:0/3133416226 conn(0x55e50fe31ec0 msgr2=0x55e50fe37bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.988+0000 7f941a000640 0 client.26120 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.988+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 shutdown_connections 2024-04-25T18:54:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.988+0000 7f943a00b740 1 -- 172.21.15.63:0/3133416226 wait complete. 2024-04-25T18:54:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:18.988+0000 7f943a00b740 20 client.26120 trim_cache size 0 max 16384 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.005+0000 7f943a00b740 1 Processor -- start 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.005+0000 7f943a00b740 1 -- start start 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 0x55e50fe4c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe37a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94301b2680 con 0x55e50fe359b0 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94301afef0 con 0x55e50fe40b90 2024-04-25T18:54:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f943a00b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94301add60 con 0x55e50fe362d0 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f9435200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f942f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe37a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f942fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 0x55e50fe4c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f9435200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60650/0 (socket says 172.21.15.63:60650) 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f942f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe37a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51126/0 (socket says 172.21.15.63:51126) 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.006+0000 7f9435200640 1 -- 172.21.15.63:0/943605477 learned_addr learned my addr 172.21.15.63:0/943605477 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 -- 172.21.15.63:0/943605477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 msgr2=0x55e50fe37a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe37a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 -- 172.21.15.63:0/943605477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 msgr2=0x55e50fe58460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 -- 172.21.15.63:0/943605477 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e50fe42f80 con 0x55e50fe359b0 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f9435200640 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f942fe00640 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 0x55e50fe4c390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f94200b9580 tx=0x7f9420031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e67867bc3c6a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f9424c00640 1 -- 172.21.15.63:0/943605477 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420049070 con 0x55e50fe359b0 2024-04-25T18:54:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f9424c00640 1 -- 172.21.15.63:0/943605477 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94200f41e0 con 0x55e50fe359b0 2024-04-25T18:54:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.007+0000 7f9424c00640 1 -- 172.21.15.63:0/943605477 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94200c0eb0 con 0x55e50fe359b0 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.008+0000 7f943a00b740 1 -- 172.21.15.63:0/943605477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 msgr2=0x55e50fe4c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.008+0000 7f943a00b740 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 0x55e50fe4c390 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f94200b9580 tx=0x7f9420031400 comp rx=0 tx=0).stop 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.008+0000 7f943a00b740 1 -- 172.21.15.63:0/943605477 shutdown_connections 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.008+0000 7f943a00b740 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe40b90 0x55e50fe58460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe362d0 0x55e50fe37a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 --2- 172.21.15.63:0/943605477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe359b0 0x55e50fe4c390 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 -- 172.21.15.63:0/943605477 >> 172.21.15.63:0/943605477 conn(0x55e50fe31ec0 msgr2=0x55e50fe37360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 -- 172.21.15.63:0/943605477 shutdown_connections 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 -- 172.21.15.63:0/943605477 wait complete. 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.009+0000 7f943a00b740 1 -- start start 2024-04-25T18:54:19.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 Processor -- start 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe45ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe40b90 0x55e50feb4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94301ae3b0 con 0x55e50fe362d0 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94301af820 con 0x55e50fe359b0 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f94301aea80 con 0x55e50fe40b90 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.010+0000 7f943a00b740 10 client.0 _ll_register_callbacks cb 0x7f94200f4bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe45ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f9435200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe40b90 0x55e50feb4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54380/0 (socket says 172.21.15.63:54380) 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 -- 172.21.15.63:0/3414457070 learned_addr learned my addr 172.21.15.63:0/3414457070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 -- 172.21.15.63:0/3414457070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe40b90 msgr2=0x55e50feb4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe40b90 0x55e50feb4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 -- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 msgr2=0x55e50fe45ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe45ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 -- 172.21.15.63:0/3414457070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e50feb52e0 con 0x55e50fe362d0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f942f400640 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f943019bf70 tx=0x7f941c0108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78dec00aecd84c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.011+0000 7f941a000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c026060 con 0x55e50fe362d0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f941c020110 con 0x55e50fe362d0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e50fda9320 con 0x55e50fe362d0 2024-04-25T18:54:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f941c020310 con 0x55e50fe362d0 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f943a00b740 10 client.15945 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.012+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55e50feb6680 con 0x55e50fe362d0 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e50fda9320 con 0x55e50fe362d0 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f941c01e3f0 con 0x55e50fe362d0 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 10 client.15945 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 20 client.15945 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50feb6d90 0x55e50feb91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 4 client.15945 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e50fda3190 con 0x55e50feb6d90 2024-04-25T18:54:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.013+0000 7f942fe00640 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50feb6d90 0x55e50feb91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.014+0000 7f942fe00640 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50feb6d90 0x55e50feb91b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=467649b7e5eb846a server_cookie=b59008372fd98df9 in_seq=0 out_seq=0 2024-04-25T18:54:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.014+0000 7f941a000640 10 client.15945 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f941a000640 1 -- 172.21.15.63:0/3414457070 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e50fda3190 con 0x55e50feb6d90 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f941a000640 10 client.15945 handle_command_reply: tid=0 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 client.15945 shutdown 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 msgr2=0x55e50fe46010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f943019bf70 tx=0x7f941c0108c0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 shutdown_connections 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e50feb6d90 0x55e50feb91b0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e50fe40b90 0x55e50feb4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e50fe362d0 0x55e50fe46010 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 --2- 172.21.15.63:0/3414457070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e50fe359b0 0x55e50fe45ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f941a000640 0 client.15945 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 >> 172.21.15.63:0/3414457070 conn(0x55e50fe31ec0 msgr2=0x55e50fe37bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 shutdown_connections 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.038+0000 7f943a00b740 1 -- 172.21.15.63:0/3414457070 wait complete. 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f943a00b740 20 client.15945 trim_cache size 0 max 16384 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:54:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 msgr2=0x7f94140a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 0x7f94140a61d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9428005fd0 tx=0x7f9428009500 comp rx=0 tx=0).stop 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 msgr2=0x7f94301a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9420005ab0 tx=0x7f9420003c10 comp rx=0 tx=0).stop 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 shutdown_connections 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f94140a3d10 0x7f94140a61d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f943019a720 0x7f943019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f94301070b0 0x7f94301a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 --2- 172.21.15.63:0/3780262473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9430101fe0 0x7f94301a0660 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.039+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 >> 172.21.15.63:0/3780262473 conn(0x7f94300fd8a0 msgr2=0x7f94300fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.040+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 shutdown_connections 2024-04-25T18:54:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:19.040+0000 7f9435c00640 1 -- 172.21.15.63:0/3780262473 wait complete. 2024-04-25T18:54:19.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.109+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a7002e0a0 con 0x7f8a70074c70 2024-04-25T18:54:19.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.110+0000 7f8a6e000700 1 -- 172.21.15.63:0/1667812526 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11871 (secure 0 0 0) 0x7f8a6409c940 con 0x7f8a70074c70 2024-04-25T18:54:19.114 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:54:19.114 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:53:53.997852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 msgr2=0x7f8a580a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 0x7f8a580a6320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8a701042d0 tx=0x7f8a5c009310 comp rx=0 tx=0).stop 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 msgr2=0x7f8a70102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400e9d0 tx=0x7f8a6400ece0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 shutdown_connections 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8a580a3e60 0x7f8a580a6320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8a70074c70 0x7f8a70102b60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a7010f440 0x7f8a701035e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 --2- 172.21.15.63:0/1667812526 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a70075590 0x7f8a701030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:19.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 >> 172.21.15.63:0/1667812526 conn(0x7f8a70079ba0 msgr2=0x7f8a70100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:19.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 shutdown_connections 2024-04-25T18:54:19.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:19.115+0000 7f8a78600700 1 -- 172.21.15.63:0/1667812526 wait complete. 2024-04-25T18:54:19.119 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 34 2024-04-25T18:54:19.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:19 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1667812526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:54:19.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.02 MB/sec execute 258 sec latency 22104.239 ms 2024-04-25T18:54:19.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:19 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1667812526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:54:19.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:19 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1667812526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:54:19.953 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id c02b8367 --release --await' 2024-04-25T18:54:19.953 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id c02b8367 --release --await 2024-04-25T18:54:20.425 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:54:20.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 4.01 MB/sec execute 259 sec latency 23104.305 ms 2024-04-25T18:54:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:20 smithi114 ceph-mon[32988]: pgmap v241: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:20.577 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:20 smithi063 ceph-mon[28006]: pgmap v241: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:20 smithi070 ceph-mon[32820]: pgmap v241: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:21.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 3.99 MB/sec execute 260 sec latency 24104.450 ms 2024-04-25T18:54:22.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 257728 3.98 MB/sec execute 261 sec latency 25104.519 ms 2024-04-25T18:54:22.494 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:22 smithi063 ceph-mon[28006]: pgmap v242: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:22.506 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:22 smithi070 ceph-mon[32820]: pgmap v242: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:22 smithi114 ceph-mon[32988]: pgmap v242: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 -- 172.21.15.63:0/3639114684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 msgr2=0x7fbf08108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/3639114684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf08108e10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098f0 tx=0x7fbef8009c00 comp rx=0 tx=0).stop 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 -- 172.21.15.63:0/3639114684 shutdown_connections 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/3639114684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806b0f0 0x7fbf0810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/3639114684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf08108e10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/3639114684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806a750 0x7fbf0806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 -- 172.21.15.63:0/3639114684 >> 172.21.15.63:0/3639114684 conn(0x7fbf080788f0 msgr2=0x7fbf08078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 -- 172.21.15.63:0/3639114684 shutdown_connections 2024-04-25T18:54:22.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.857+0000 7fbf0ea00700 1 -- 172.21.15.63:0/3639114684 wait complete. 2024-04-25T18:54:22.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.858+0000 7fbf0ea00700 1 Processor -- start 2024-04-25T18:54:22.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.858+0000 7fbf0ea00700 1 -- start start 2024-04-25T18:54:22.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 0x7fbf0819b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf0819fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08112c00 con 0x7fbf0806b0f0 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08112a80 con 0x7fbf08108a30 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf0ea00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08112d80 con 0x7fbf0806a750 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf07400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf07400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54396/0 (socket says 172.21.15.63:54396) 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf07400700 1 -- 172.21.15.63:0/566192559 learned_addr learned my addr 172.21.15.63:0/566192559 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.859+0000 7fbf07e00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 0x7fbf0819b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf0819fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 -- 172.21.15.63:0/566192559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 msgr2=0x7fbf0819fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf0819fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 -- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 msgr2=0x7fbf0819b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 -- 172.21.15.63:0/566192559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf081a03c0 con 0x7fbf0806a750 2024-04-25T18:54:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf0819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07400700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf07e00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 0x7fbf0819b3b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c010 tx=0x7fbef800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.860+0000 7fbf04c00700 1 -- 172.21.15.63:0/566192559 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80051c0 con 0x7fbf0806a750 2024-04-25T18:54:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.861+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf081a0650 con 0x7fbf0806a750 2024-04-25T18:54:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.861+0000 7fbf04c00700 1 -- 172.21.15.63:0/566192559 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbef800b560 con 0x7fbf0806a750 2024-04-25T18:54:22.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.861+0000 7fbf04c00700 1 -- 172.21.15.63:0/566192559 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80039e0 con 0x7fbf0806a750 2024-04-25T18:54:22.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.861+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf081a7f30 con 0x7fbf0806a750 2024-04-25T18:54:22.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.864+0000 7fbf04c00700 1 -- 172.21.15.63:0/566192559 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fbef800b930 con 0x7fbf0806a750 2024-04-25T18:54:22.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.865+0000 7fbf04c00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 0x7fbeec0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.865+0000 7fbf04c00700 1 -- 172.21.15.63:0/566192559 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fbef80d16a0 con 0x7fbf0806a750 2024-04-25T18:54:22.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf1944fb80 1 Processor -- start 2024-04-25T18:54:22.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf07400700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 0x7fbeec0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf1944fb80 1 -- start start 2024-04-25T18:54:22.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18a4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.866+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b18a50f0 0x55c1b18a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1b17ecf90 con 0x55c1b189a240 2024-04-25T18:54:22.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b1793830 con 0x55c1b18a50f0 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1b17e8f50 con 0x55c1b1899920 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51172/0 (socket says 172.21.15.63:51172) 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 -- 172.21.15.63:0/1734789084 learned_addr learned my addr 172.21.15.63:0/1734789084 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07400700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 0x7fbeec0a6270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009750 tx=0x7fbef4009310 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b18a50f0 0x55c1b18a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07400700 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18a4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 -- 172.21.15.63:0/1734789084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b18a50f0 msgr2=0x55c1b18a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b18a50f0 0x55c1b18a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 -- 172.21.15.63:0/1734789084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 msgr2=0x55c1b18a4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18a4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.867+0000 7fbf07e00700 1 -- 172.21.15.63:0/1734789084 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b1789780 con 0x55c1b1899920 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.868+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbef800bc80 tx=0x7fbef8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=929d0290b9b71e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.868+0000 7fbefd600700 1 -- 172.21.15.63:0/1734789084 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef809d630 con 0x55c1b1899920 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.868+0000 7fbefd600700 1 -- 172.21.15.63:0/1734789084 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbef80a19e0 con 0x55c1b1899920 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.868+0000 7fbefd600700 1 -- 172.21.15.63:0/1734789084 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef8021380 con 0x55c1b1899920 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1734789084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 msgr2=0x55c1b1899d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbef800bc80 tx=0x7fbef8009260 comp rx=0 tx=0).stop 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1734789084 shutdown_connections 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b1899920 0x55c1b1899d00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b18a50f0 0x55c1b18a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1734789084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18a4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1734789084 >> 172.21.15.63:0/1734789084 conn(0x55c1b17c86a0 msgr2=0x55c1b1895dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1734789084 shutdown_connections 2024-04-25T18:54:22.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1734789084 wait complete. 2024-04-25T18:54:22.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.869+0000 7fbf1944fb80 1 -- start start 2024-04-25T18:54:22.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 Processor -- start 2024-04-25T18:54:22.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b1899920 0x55c1b18a4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1b18a7a20 con 0x55c1b1899920 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b17a6960 con 0x55c1b189a240 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1b18a7ba0 con 0x55c1b18a50f0 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.870+0000 7fbf1944fb80 10 client.0 _ll_register_callbacks cb 0x55c1b1785820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b1899920 0x55c1b18a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60692/0 (socket says 172.21.15.63:60692) 2024-04-25T18:54:22.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 -- 172.21.15.63:0/3198527858 learned_addr learned my addr 172.21.15.63:0/3198527858 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 -- 172.21.15.63:0/3198527858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 msgr2=0x55c1b18a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 -- 172.21.15.63:0/3198527858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b1899920 msgr2=0x55c1b18a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b1899920 0x55c1b18a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 -- 172.21.15.63:0/3198527858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b173a9d0 con 0x55c1b189a240 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.871+0000 7fbf07400700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbef401d9d0 tx=0x7fbef40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffc3759473aed2e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbef2000700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef4004020 con 0x55c1b189a240 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbef80a19e0 con 0x55c1b189a240 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fbef40041c0 con 0x55c1b189a240 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbef40043c0 con 0x55c1b189a240 2024-04-25T18:54:22.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbf1944fb80 10 client.26138 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:22.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c1b1783bf0 con 0x55c1b189a240 2024-04-25T18:54:22.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.872+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbef80a19e0 con 0x55c1b189a240 2024-04-25T18:54:22.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fbef4025070 con 0x55c1b189a240 2024-04-25T18:54:22.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 10 client.26138 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:22.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 10 client.26138 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 20 client.26138 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b18b4710 0x55c1b18b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 4 client.26138 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1b17fa780 con 0x55c1b18b4710 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.873+0000 7fbf07e00700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b18b4710 0x55c1b18b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.874+0000 7fbf07e00700 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b18b4710 0x55c1b18b6b40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=967adfdee016395c server_cookie=285803b533575e00 in_seq=0 out_seq=0 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.874+0000 7fbef2000700 10 client.26138 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbef2000700 1 -- 172.21.15.63:0/3198527858 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1b17fa780 con 0x55c1b18b4710 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbef2000700 10 client.26138 handle_command_reply: tid=0 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 client.26138 shutdown 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 msgr2=0x55c1b18a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbef401d9d0 tx=0x7fbef40047c0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 shutdown_connections 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b18b4710 0x55c1b18b6b40 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b1899920 0x55c1b18a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbef2000700 0 client.26138 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/3198527858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b189a240 0x55c1b18a1b70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 >> 172.21.15.63:0/3198527858 conn(0x55c1b17c86a0 msgr2=0x55c1b18959c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 shutdown_connections 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 1 -- 172.21.15.63:0/3198527858 wait complete. 2024-04-25T18:54:22.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.875+0000 7fbf1944fb80 20 client.26138 trim_cache size 0 max 16384 2024-04-25T18:54:22.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 Processor -- start 2024-04-25T18:54:22.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 -- start start 2024-04-25T18:54:22.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18bc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18bc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081b1ec0 con 0x55c1b189a240 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081af6f0 con 0x55c1b1899920 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.897+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081ad560 con 0x55c1b18a50f0 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60704/0 (socket says 172.21.15.63:60704) 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf07e00700 1 -- 172.21.15.63:0/1345646799 learned_addr learned my addr 172.21.15.63:0/1345646799 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf07400700 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18bc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 -- 172.21.15.63:0/1345646799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 msgr2=0x55c1b18b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18b08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 -- 172.21.15.63:0/1345646799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 msgr2=0x55c1b18bc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18bc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.898+0000 7fbf0ca00700 1 -- 172.21.15.63:0/1345646799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b18a78a0 con 0x55c1b18a50f0 2024-04-25T18:54:22.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.899+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18bc990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf000075b0 tx=0x7fbf0000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5b4a0447472866 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.899+0000 7fbefcc00700 1 -- 172.21.15.63:0/1345646799 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00003c00 con 0x55c1b18a50f0 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.899+0000 7fbefcc00700 1 -- 172.21.15.63:0/1345646799 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf00007e40 con 0x55c1b18a50f0 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.899+0000 7fbefcc00700 1 -- 172.21.15.63:0/1345646799 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00017760 con 0x55c1b18a50f0 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1345646799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 msgr2=0x55c1b18bc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18bc990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf000075b0 tx=0x7fbf0000b040 comp rx=0 tx=0).stop 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1345646799 shutdown_connections 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b18bc990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18b08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1345646799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18bc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1345646799 >> 172.21.15.63:0/1345646799 conn(0x55c1b17c86a0 msgr2=0x55c1b18a1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1345646799 shutdown_connections 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.900+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1345646799 wait complete. 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 -- start start 2024-04-25T18:54:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 Processor -- start 2024-04-25T18:54:22.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b191acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081adbb0 con 0x55c1b189a240 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081af020 con 0x55c1b1899920 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.901+0000 7fbf1944fb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fbf081ae280 con 0x55c1b18a50f0 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf1944fb80 10 client.0 _ll_register_callbacks cb 0x55c1b1785820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18aa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54432/0 (socket says 172.21.15.63:54432) 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07400700 1 -- 172.21.15.63:0/1305454189 learned_addr learned my addr 172.21.15.63:0/1305454189 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf0ca00700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b191acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 -- 172.21.15.63:0/1305454189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 msgr2=0x55c1b191acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b191acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 -- 172.21.15.63:0/1305454189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 msgr2=0x55c1b18aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.902+0000 7fbf07e00700 1 -- 172.21.15.63:0/1305454189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b191b210 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbf07e00700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18a9ba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098c0 tx=0x7fbef800bcd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27b2104197f5b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef80a19e0 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c1b17f4820 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fbef8099ca0 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fbef80bc340 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbf1944fb80 10 client.26144 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55c1b191c3b0 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c1b17f4820 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.903+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fbef80bc020 con 0x55c1b1899920 2024-04-25T18:54:22.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 10 client.26144 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 10 client.26144 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 20 client.26144 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b191c730 0x55c1b191eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 4 client.26144 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c02b8367", "release": true, "await": true}] 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c02b8367", "release": true, "await": true}) -- 0x55c1b166d350 con 0x55c1b191c730 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf07400700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b191c730 0x55c1b191eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:22.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbf07400700 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b191c730 0x55c1b191eb50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61333cf2d101618b server_cookie=daaf5457eedada3e in_seq=0 out_seq=0 2024-04-25T18:54:22.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.904+0000 7fbef2000700 10 client.26144 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:22.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.912+0000 7fbef2000700 1 -- 172.21.15.63:0/1305454189 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c1b166d350 con 0x55c1b191c730 2024-04-25T18:54:22.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.912+0000 7fbef2000700 10 client.26144 handle_command_reply: tid=0 2024-04-25T18:54:22.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.912+0000 7fbf1944fb80 1 client.26144 shutdown 2024-04-25T18:54:22.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.912+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 msgr2=0x55c1b18a9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18a9ba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbef80098c0 tx=0x7fbef800bcd0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 shutdown_connections 2024-04-25T18:54:22.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b191c730 0x55c1b191eb50 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b18a50f0 0x55c1b191acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbef2000700 0 client.26144 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b1899920 0x55c1b18a9ba0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 --2- 172.21.15.63:0/1305454189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b189a240 0x55c1b18aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 >> 172.21.15.63:0/1305454189 conn(0x55c1b17c86a0 msgr2=0x55c1b18a33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 shutdown_connections 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 1 -- 172.21.15.63:0/1305454189 wait complete. 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.913+0000 7fbf1944fb80 20 client.26144 trim_cache size 0 max 16384 2024-04-25T18:54:22.916 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 34, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 24, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "c02b8367": { 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 24, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 77.4, 2024-04-25T18:54:22.917 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:54:22.918 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 msgr2=0x7fbeec0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 0x7fbeec0a6270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009750 tx=0x7fbef4009310 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 msgr2=0x7fbf0819b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 0x7fbf0819b3b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbef800c010 tx=0x7fbef800fa00 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 shutdown_connections 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fbeec0a3db0 0x7fbeec0a6270 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fbf0806a750 0x7fbf0819b3b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf08108a30 0x7fbf0819fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 --2- 172.21.15.63:0/566192559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf0806b0f0 0x7fbf0819b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:22.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.914+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 >> 172.21.15.63:0/566192559 conn(0x7fbf080788f0 msgr2=0x7fbf0810def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:22.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.915+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 shutdown_connections 2024-04-25T18:54:22.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:22.915+0000 7fbf0ea00700 1 -- 172.21.15.63:0/566192559 wait complete. 2024-04-25T18:54:23.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 258419 3.98 MB/sec execute 262 sec latency 25537.760 ms 2024-04-25T18:54:23.902 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c02b8367', seconds elapsed: 34.0 2024-04-25T18:54:24.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 259506 3.99 MB/sec execute 263 sec latency 32.731 ms 2024-04-25T18:54:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:24 smithi114 ceph-mon[32988]: pgmap v243: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:24 smithi063 ceph-mon[28006]: pgmap v243: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:24 smithi070 ceph-mon[32820]: pgmap v243: 97 pgs: 97 active+clean; 85 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 978 KiB/s wr, 48 op/s 2024-04-25T18:54:25.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 260618 4.00 MB/sec execute 264 sec latency 25.554 ms 2024-04-25T18:54:26.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 261991 4.01 MB/sec execute 265 sec latency 21.112 ms 2024-04-25T18:54:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:26 smithi114 ceph-mon[32988]: pgmap v244: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-25T18:54:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:26 smithi063 ceph-mon[28006]: pgmap v244: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-25T18:54:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:26 smithi070 ceph-mon[32820]: pgmap v244: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-04-25T18:54:27.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 263124 4.02 MB/sec execute 266 sec latency 22.417 ms 2024-04-25T18:54:28.479 INFO:tasks.workunit.client.0.smithi063.stdout: 1 264085 4.02 MB/sec execute 267 sec latency 21.921 ms 2024-04-25T18:54:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:28 smithi114 ceph-mon[32988]: pgmap v245: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-25T18:54:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:28 smithi063 ceph-mon[28006]: pgmap v245: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-25T18:54:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:28 smithi070 ceph-mon[32820]: pgmap v245: 97 pgs: 97 active+clean; 86 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 1.1 MiB/s wr, 55 op/s 2024-04-25T18:54:29.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 265491 4.04 MB/sec execute 268 sec latency 23.867 ms 2024-04-25T18:54:30.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 266652 4.05 MB/sec execute 269 sec latency 13.736 ms 2024-04-25T18:54:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:30 smithi114 ceph-mon[32988]: pgmap v246: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-04-25T18:54:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:30 smithi063 ceph-mon[28006]: pgmap v246: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-04-25T18:54:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:30 smithi070 ceph-mon[32820]: pgmap v246: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-04-25T18:54:31.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 267655 4.05 MB/sec execute 270 sec latency 25.433 ms 2024-04-25T18:54:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 -- 172.21.15.63:0/21405025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074380 msgr2=0x7fdc60074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 --2- 172.21.15.63:0/21405025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074380 0x7fdc60074760 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009860 tx=0x7fdc5002a3d0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 -- 172.21.15.63:0/21405025 shutdown_connections 2024-04-25T18:54:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 --2- 172.21.15.63:0/21405025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc6010e5a0 0x7fdc601109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 --2- 172.21.15.63:0/21405025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc60074ca0 0x7fdc6010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 --2- 172.21.15.63:0/21405025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074380 0x7fdc60074760 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.190+0000 7fdc65800640 1 -- 172.21.15.63:0/21405025 >> 172.21.15.63:0/21405025 conn(0x7fdc600fd890 msgr2=0x7fdc600ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.191+0000 7fdc65800640 1 -- 172.21.15.63:0/21405025 shutdown_connections 2024-04-25T18:54:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.191+0000 7fdc65800640 1 -- 172.21.15.63:0/21405025 wait complete. 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 Processor -- start 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 -- start start 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc60074380 0x7fdc601a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 0x7fdc6019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60114220 con 0x7fdc60074380 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601140a0 con 0x7fdc60074ca0 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc65800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601143a0 con 0x7fdc6010e5a0 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc64e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 0x7fdc6019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59944/0 (socket says 172.21.15.63:59944) 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc5f400640 1 -- 172.21.15.63:0/3387112344 learned_addr learned my addr 172.21.15.63:0/3387112344 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.192+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc60074380 0x7fdc601a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 -- 172.21.15.63:0/3387112344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 msgr2=0x7fdc6019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 0x7fdc6019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 -- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc60074380 msgr2=0x7fdc601a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc60074380 0x7fdc601a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 -- 172.21.15.63:0/3387112344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6019ad60 con 0x7fdc60074ca0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc64e00640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 0x7fdc6019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e9c0 tx=0x7fdc4c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5cc00640 1 -- 172.21.15.63:0/3387112344 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c00cde0 con 0x7fdc60074ca0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5cc00640 1 -- 172.21.15.63:0/3387112344 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc4c0035f0 con 0x7fdc60074ca0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc5cc00640 1 -- 172.21.15.63:0/3387112344 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c010690 con 0x7fdc60074ca0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.193+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6019b050 con 0x7fdc60074ca0 2024-04-25T18:54:32.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.194+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc601a72f0 con 0x7fdc60074ca0 2024-04-25T18:54:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.197+0000 7fdc69d90740 1 Processor -- start 2024-04-25T18:54:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.197+0000 7fdc69d90740 1 -- start start 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c53091650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53093f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557c52fec900 con 0x557c53091b90 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557c52f92ef0 con 0x557c530872d0 2024-04-25T18:54:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc69d90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c52fe88c0 con 0x557c530869b0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33254/0 (socket says 172.21.15.63:33254) 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c53091650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 -- 172.21.15.63:0/567197904 learned_addr learned my addr 172.21.15.63:0/567197904 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc5cc00640 1 -- 172.21.15.63:0/3387112344 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc4c004000 con 0x7fdc60074ca0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 -- 172.21.15.63:0/567197904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 msgr2=0x557c53091650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c53091650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 -- 172.21.15.63:0/567197904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 msgr2=0x557c53093f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53093f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.198+0000 7fdc64e00640 1 -- 172.21.15.63:0/567197904 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c52cff210 con 0x557c530869b0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.199+0000 7fdc64e00640 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011790 tx=0x7fdc58011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b4d79ad09c0251 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.199+0000 7fdc55600640 1 -- 172.21.15.63:0/567197904 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58003c30 con 0x557c530869b0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.199+0000 7fdc55600640 1 -- 172.21.15.63:0/567197904 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5800fe50 con 0x557c530869b0 2024-04-25T18:54:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.199+0000 7fdc55600640 1 -- 172.21.15.63:0/567197904 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc580193e0 con 0x557c530869b0 2024-04-25T18:54:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 -- 172.21.15.63:0/567197904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 msgr2=0x557c53086d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011790 tx=0x7fdc58011c50 comp rx=0 tx=0).stop 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 -- 172.21.15.63:0/567197904 shutdown_connections 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c53091650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53093f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc5cc00640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 0x7fdc440a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 --2- 172.21.15.63:0/567197904 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c53086d90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 -- 172.21.15.63:0/567197904 >> 172.21.15.63:0/567197904 conn(0x557c53082ec0 msgr2=0x557c530832a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 -- 172.21.15.63:0/567197904 shutdown_connections 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc5cc00640 1 -- 172.21.15.63:0/3387112344 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fdc4c0d0b40 con 0x7fdc60074ca0 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc69d90740 1 -- 172.21.15.63:0/567197904 wait complete. 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.200+0000 7fdc64e00640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 0x7fdc440a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 -- start start 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 Processor -- start 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc64e00640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 0x7fdc440a6250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011790 tx=0x7fdc58005bd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 0x557c5308f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c53091b90 0x557c5308f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557c530947c0 con 0x557c530872d0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557c52fa6310 con 0x557c53091b90 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.201+0000 7fdc69d90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557c53094940 con 0x557c530869b0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc69d90740 10 client.0 _ll_register_callbacks cb 0x557c52f74cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33268/0 (socket says 172.21.15.63:33268) 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc64e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c53091b90 0x557c5308f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 -- 172.21.15.63:0/975126549 learned_addr learned my addr 172.21.15.63:0/975126549 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5f400640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 0x557c5308f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 -- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c53091b90 msgr2=0x557c5308f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c53091b90 0x557c5308f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 -- 172.21.15.63:0/975126549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 msgr2=0x557c5308f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 0x557c5308f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 -- 172.21.15.63:0/975126549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c52dcb620 con 0x557c530869b0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5f400640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 0x557c5308f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009860 tx=0x7fdc50030bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edf3c5358665282e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.202+0000 7fdc4a000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50009450 con 0x557c530869b0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5800fe50 con 0x557c530869b0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fdc50004050 con 0x557c530869b0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdc50004250 con 0x557c530869b0 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc69d90740 10 client.26179 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557c530a0a50 con 0x557c530869b0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.203+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5800fe50 con 0x557c530869b0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fdc50004960 con 0x557c530869b0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 10 client.26179 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 10 client.26179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 20 client.26179 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c530a1f10 0x557c530a4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 4 client.26179 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c530008b0 con 0x557c530a1f10 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.204+0000 7fdc5f400640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c530a1f10 0x557c530a4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.205+0000 7fdc5f400640 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c530a1f10 0x557c530a4320 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b9cb24b0a893ed6 server_cookie=3f054c136d76bc14 in_seq=0 out_seq=0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.205+0000 7fdc4a000640 10 client.26179 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc4a000640 1 -- 172.21.15.63:0/975126549 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557c530008b0 con 0x557c530a1f10 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc4a000640 10 client.26179 handle_command_reply: tid=0 2024-04-25T18:54:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 client.26179 shutdown 2024-04-25T18:54:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 msgr2=0x557c5308eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc50009860 tx=0x7fdc50030bb0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 shutdown_connections 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c530a1f10 0x557c530a4320 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c53091b90 0x557c5308f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c530872d0 0x557c5308f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 --2- 172.21.15.63:0/975126549 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c5308eac0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 >> 172.21.15.63:0/975126549 conn(0x557c53082ec0 msgr2=0x557c53088bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc4a000640 0 client.26179 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 shutdown_connections 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 1 -- 172.21.15.63:0/975126549 wait complete. 2024-04-25T18:54:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.206+0000 7fdc69d90740 20 client.26179 trim_cache size 0 max 16384 2024-04-25T18:54:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 Processor -- start 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 -- start start 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c530827a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c530a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601b2700 con 0x557c53091b90 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601aff70 con 0x557c530872d0 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc69d90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601adde0 con 0x557c530869b0 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc64e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c530a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.224+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c530827a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59958/0 (socket says 172.21.15.63:59958) 2024-04-25T18:54:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5f400640 1 -- 172.21.15.63:0/3612116107 learned_addr learned my addr 172.21.15.63:0/3612116107 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 -- 172.21.15.63:0/3612116107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 msgr2=0x557c5309d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 -- 172.21.15.63:0/3612116107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 msgr2=0x557c530a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c530a9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 -- 172.21.15.63:0/3612116107 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c5307e820 con 0x557c530869b0 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc64e00640 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c530a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.225+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c530827a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc50000c00 tx=0x7fdc50005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e26560f95236ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.226+0000 7fdc54c00640 1 -- 172.21.15.63:0/3612116107 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50053040 con 0x557c530869b0 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.226+0000 7fdc54c00640 1 -- 172.21.15.63:0/3612116107 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc500035f0 con 0x557c530869b0 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.226+0000 7fdc54c00640 1 -- 172.21.15.63:0/3612116107 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50005200 con 0x557c530869b0 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.226+0000 7fdc69d90740 1 -- 172.21.15.63:0/3612116107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 msgr2=0x557c530827a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c530827a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc50000c00 tx=0x7fdc50005a90 comp rx=0 tx=0).stop 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 -- 172.21.15.63:0/3612116107 shutdown_connections 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530872d0 0x557c5309d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c530a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3612116107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530869b0 0x557c530827a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 -- 172.21.15.63:0/3612116107 >> 172.21.15.63:0/3612116107 conn(0x557c53082ec0 msgr2=0x557c53089e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 -- 172.21.15.63:0/3612116107 shutdown_connections 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 -- 172.21.15.63:0/3612116107 wait complete. 2024-04-25T18:54:32.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 -- start start 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.227+0000 7fdc69d90740 1 Processor -- start 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53098da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601ae430 con 0x557c53091b90 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601af8a0 con 0x557c530869b0 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601aeb00 con 0x557c530872d0 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc69d90740 10 client.0 _ll_register_callbacks cb 0x7fdc50030880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc5f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59960/0 (socket says 172.21.15.63:59960) 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc5fe00640 1 -- 172.21.15.63:0/3320797559 learned_addr learned my addr 172.21.15.63:0/3320797559 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33274/0 (socket says 172.21.15.63:33274) 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.228+0000 7fdc64e00640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53098da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 -- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 msgr2=0x557c53099de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 -- 172.21.15.63:0/3320797559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 msgr2=0x557c53098da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53098da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 -- 172.21.15.63:0/3320797559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c53099360 con 0x557c530872d0 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc5f400640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c003e20 tx=0x7fdc4c003d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f32bc55fb0bdb1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:32.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc4a000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0a0df0 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7fdc4c003aa0 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc500035f0 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fdc4c0a56b0 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.229+0000 7fdc69d90740 10 client.26185 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x557c53108270 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc500035f0 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fdc4c0ce500 con 0x557c530872d0 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 10 client.26185 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 10 client.26185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 20 client.26185 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c531089d0 0x557c5310adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 4 client.26185 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:54:32.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557c52ffa320 con 0x557c531089d0 2024-04-25T18:54:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.230+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c531089d0 0x557c5310adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.231+0000 7fdc5fe00640 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c531089d0 0x557c5310adf0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e56f694c553f408 server_cookie=fce12b1552264a3e in_seq=0 out_seq=0 2024-04-25T18:54:32.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.231+0000 7fdc4a000640 10 client.26185 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc4a000640 1 -- 172.21.15.63:0/3320797559 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557c52ffa320 con 0x557c531089d0 2024-04-25T18:54:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc4a000640 10 client.26185 handle_command_reply: tid=0 2024-04-25T18:54:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 client.26185 shutdown 2024-04-25T18:54:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 msgr2=0x557c53098860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c003e20 tx=0x7fdc4c003d70 comp rx=0 tx=0).stop 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 shutdown_connections 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557c531089d0 0x557c5310adf0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557c53091b90 0x557c53098da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557c530872d0 0x557c53098860 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc4a000640 0 client.26185 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 --2- 172.21.15.63:0/3320797559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557c530869b0 0x557c53099de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.272+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 >> 172.21.15.63:0/3320797559 conn(0x557c53082ec0 msgr2=0x557c530902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.273+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 shutdown_connections 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.273+0000 7fdc69d90740 1 -- 172.21.15.63:0/3320797559 wait complete. 2024-04-25T18:54:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.273+0000 7fdc69d90740 20 client.26185 trim_cache size 0 max 16384 2024-04-25T18:54:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:54:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:54:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:54:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:54:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 msgr2=0x7fdc440a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 0x7fdc440a6250 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011790 tx=0x7fdc58005bd0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 msgr2=0x7fdc601a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00e9c0 tx=0x7fdc4c00ee80 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 shutdown_connections 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc440a3d90 0x7fdc440a6250 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc60074ca0 0x7fdc601a0970 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc6010e5a0 0x7fdc6019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 --2- 172.21.15.63:0/3387112344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc60074380 0x7fdc601a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 >> 172.21.15.63:0/3387112344 conn(0x7fdc600fd890 msgr2=0x7fdc600ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 shutdown_connections 2024-04-25T18:54:32.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:32.274+0000 7fdc65800640 1 -- 172.21.15.63:0/3387112344 wait complete. 2024-04-25T18:54:32.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 269267 4.06 MB/sec execute 271 sec latency 36.115 ms 2024-04-25T18:54:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:32 smithi114 ceph-mon[32988]: pgmap v247: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:32 smithi063 ceph-mon[28006]: pgmap v247: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:32 smithi070 ceph-mon[32820]: pgmap v247: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:33.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 270335 4.07 MB/sec execute 272 sec latency 23.330 ms 2024-04-25T18:54:34.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 271306 4.08 MB/sec execute 273 sec latency 21.402 ms 2024-04-25T18:54:34.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:34 smithi114 ceph-mon[32988]: pgmap v248: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:34 smithi063 ceph-mon[28006]: pgmap v248: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:34 smithi070 ceph-mon[32820]: pgmap v248: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.4 MiB/s wr, 104 op/s 2024-04-25T18:54:35.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 273137 4.10 MB/sec execute 274 sec latency 15.875 ms 2024-04-25T18:54:36.480 INFO:tasks.workunit.client.0.smithi063.stdout: 1 274025 4.10 MB/sec execute 275 sec latency 27.045 ms 2024-04-25T18:54:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:36 smithi114 ceph-mon[32988]: pgmap v249: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 196 op/s 2024-04-25T18:54:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:36 smithi063 ceph-mon[28006]: pgmap v249: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 196 op/s 2024-04-25T18:54:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:36 smithi070 ceph-mon[32820]: pgmap v249: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 196 op/s 2024-04-25T18:54:37.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 275292 4.10 MB/sec execute 276 sec latency 25.904 ms 2024-04-25T18:54:38.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 276702 4.13 MB/sec execute 277 sec latency 17.000 ms 2024-04-25T18:54:38.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:38 smithi114 ceph-mon[32988]: pgmap v250: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-25T18:54:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:38 smithi063 ceph-mon[28006]: pgmap v250: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-25T18:54:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:38 smithi070 ceph-mon[32820]: pgmap v250: 97 pgs: 97 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-25T18:54:39.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 277417 4.12 MB/sec execute 278 sec latency 28.733 ms 2024-04-25T18:54:40.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 278352 4.12 MB/sec execute 279 sec latency 21.770 ms 2024-04-25T18:54:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:40 smithi114 ceph-mon[32988]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-04-25T18:54:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:40 smithi063 ceph-mon[28006]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-04-25T18:54:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:40 smithi070 ceph-mon[32820]: pgmap v251: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.8 MiB/s wr, 220 op/s 2024-04-25T18:54:41.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 279691 4.14 MB/sec execute 280 sec latency 22.565 ms 2024-04-25T18:54:42.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 280728 4.14 MB/sec execute 281 sec latency 27.413 ms 2024-04-25T18:54:42.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:42 smithi114 ceph-mon[32988]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:42.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:42 smithi063 ceph-mon[28006]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:42 smithi070 ceph-mon[32820]: pgmap v252: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:43.481 INFO:tasks.workunit.client.0.smithi063.stdout: 1 281937 4.15 MB/sec execute 282 sec latency 20.509 ms 2024-04-25T18:54:44.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 283749 4.18 MB/sec execute 283 sec latency 17.658 ms 2024-04-25T18:54:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:44 smithi114 ceph-mon[32988]: pgmap v253: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:44 smithi063 ceph-mon[28006]: pgmap v253: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:44 smithi070 ceph-mon[32820]: pgmap v253: 97 pgs: 97 active+clean; 87 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:54:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 -- 172.21.15.63:0/3515665019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 msgr2=0x7f28781023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 --2- 172.21.15.63:0/3515665019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781023a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2868009900 tx=0x7f286802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 -- 172.21.15.63:0/3515665019 shutdown_connections 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 --2- 172.21.15.63:0/3515665019 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2878107090 0x7f287810ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 --2- 172.21.15.63:0/3515665019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878102970 0x7f28781069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 --2- 172.21.15.63:0/3515665019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781023a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 -- 172.21.15.63:0/3515665019 >> 172.21.15.63:0/3515665019 conn(0x7f28780fd8a0 msgr2=0x7f28780ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 -- 172.21.15.63:0/3515665019 shutdown_connections 2024-04-25T18:54:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.429+0000 7f287d600640 1 -- 172.21.15.63:0/3515665019 wait complete. 2024-04-25T18:54:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.430+0000 7f287d600640 1 Processor -- start 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.430+0000 7f287d600640 1 -- start start 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878107090 0x7f2878198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 0x7f287819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2878112040 con 0x7f2878107090 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2878111ec0 con 0x7f2878101fc0 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287d600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28781121c0 con 0x7f287819cea0 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f2877e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f287cc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 0x7f287819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f2877e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56714/0 (socket says 172.21.15.63:56714) 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f2877e00640 1 -- 172.21.15.63:0/738817587 learned_addr learned my addr 172.21.15.63:0/738817587 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.431+0000 7f2877400640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878107090 0x7f2878198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 -- 172.21.15.63:0/738817587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 msgr2=0x7f287819d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 0x7f287819d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 -- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878107090 msgr2=0x7f2878198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878107090 0x7f2878198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 -- 172.21.15.63:0/738817587 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f287819d990 con 0x7f2878101fc0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f287cc00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 0x7f287819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2877e00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2868005ab0 tx=0x7f2868003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2874c00640 1 -- 172.21.15.63:0/738817587 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286803f070 con 0x7f2878101fc0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f287819dc20 con 0x7f2878101fc0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2874c00640 1 -- 172.21.15.63:0/738817587 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2868031b60 con 0x7f2878101fc0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f2874c00640 1 -- 172.21.15.63:0/738817587 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868035030 con 0x7f2878101fc0 2024-04-25T18:54:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.432+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28781a5440 con 0x7f2878101fc0 2024-04-25T18:54:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 Processor -- start 2024-04-25T18:54:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 -- start start 2024-04-25T18:54:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2874c00640 1 -- 172.21.15.63:0/738817587 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2868042070 con 0x7f2878101fc0 2024-04-25T18:54:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab6cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562aeaad2900 con 0x562aeab6c9b0 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562aeaa78ef0 con 0x562aeab6d2d0 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.436+0000 7f2881975740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562aeaace8c0 con 0x562aeab77b90 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab6cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56720/0 (socket says 172.21.15.63:56720) 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877e00640 1 -- 172.21.15.63:0/1226505368 learned_addr learned my addr 172.21.15.63:0/1226505368 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 -- 172.21.15.63:0/1226505368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 msgr2=0x562aeab79f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab79f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 -- 172.21.15.63:0/1226505368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 msgr2=0x562aeab77650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 -- 172.21.15.63:0/1226505368 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aea7e5210 con 0x562aeab6c9b0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877400640 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f2877e00640 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.437+0000 7f287cc00640 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab6cd90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2870005fd0 tx=0x7f2870011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c494261022508ac1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f2874c00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 0x7f285c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f286d200640 1 -- 172.21.15.63:0/1226505368 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870004d20 con 0x562aeab6c9b0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f286d200640 1 -- 172.21.15.63:0/1226505368 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287000f9a0 con 0x562aeab6c9b0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f286d200640 1 -- 172.21.15.63:0/1226505368 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870004450 con 0x562aeab6c9b0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f2874c00640 1 -- 172.21.15.63:0/738817587 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f28680f0e40 con 0x7f2878101fc0 2024-04-25T18:54:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f2877e00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 0x7f285c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f2881975740 1 -- 172.21.15.63:0/1226505368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 msgr2=0x562aeab6cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.438+0000 7f2881975740 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab6cd90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2870005fd0 tx=0x7f2870011bf0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 -- 172.21.15.63:0/1226505368 shutdown_connections 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 --2- 172.21.15.63:0/1226505368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab6cd90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 -- 172.21.15.63:0/1226505368 >> 172.21.15.63:0/1226505368 conn(0x562aeab68ec0 msgr2=0x562aeab692a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2877e00640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 0x7f285c0a6250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28680bcd80 tx=0x7f28680f48f0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 -- 172.21.15.63:0/1226505368 shutdown_connections 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 -- 172.21.15.63:0/1226505368 wait complete. 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.439+0000 7f2881975740 1 -- start start 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 Processor -- start 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab74a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 0x562aeab754f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562aeab7a7c0 con 0x562aeab6c9b0 2024-04-25T18:54:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562aeaa8c310 con 0x562aeab77b90 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562aeab7a940 con 0x562aeab6d2d0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2881975740 10 client.0 _ll_register_callbacks cb 0x562aeaa5acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2877400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2877400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34402/0 (socket says 172.21.15.63:34402) 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f2877400640 1 -- 172.21.15.63:0/1853532978 learned_addr learned my addr 172.21.15.63:0/1853532978 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.440+0000 7f287cc00640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab74a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 -- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 msgr2=0x562aeab754f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f287cc00640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 0x562aeab754f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 0x562aeab754f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 -- 172.21.15.63:0/1853532978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 msgr2=0x562aeab74a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab74a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 -- 172.21.15.63:0/1853532978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aea8b1620 con 0x562aeab6d2d0 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f287cc00640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 0x562aeab754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:54:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2877400640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2864012700 tx=0x7f2864012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c329d9e8d4c1714 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2861600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.441+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864004030 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f287000f9a0 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f28640041d0 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f28640043d0 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2881975740 10 client.26197 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562aeab8ae90 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287000f9a0 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f2864003b50 con 0x562aeab6d2d0 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2881975740 10 client.26197 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.442+0000 7f2881975740 10 client.26197 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.443+0000 7f2881975740 20 client.26197 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.443+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeab8c300 0x562aeab8e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.443+0000 7f2881975740 4 client.26197 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.443+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562aeaae68b0 con 0x562aeab8c300 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.443+0000 7f287cc00640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeab8c300 0x562aeab8e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.444+0000 7f2861600640 10 client.26197 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.444+0000 7f287cc00640 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeab8c300 0x562aeab8e6f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b6959c13587b48 server_cookie=d3346d98e140625c in_seq=0 out_seq=0 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2861600640 1 -- 172.21.15.63:0/1853532978 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562aeaae68b0 con 0x562aeab8c300 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2861600640 10 client.26197 handle_command_reply: tid=0 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 client.26197 shutdown 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 msgr2=0x562aeab74fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2864012700 tx=0x7f2864012bc0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 shutdown_connections 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeab8c300 0x562aeab8e6f0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab77b90 0x562aeab754f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab6d2d0 0x562aeab74fb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2861600640 0 client.26197 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 --2- 172.21.15.63:0/1853532978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab74a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 >> 172.21.15.63:0/1853532978 conn(0x562aeab68ec0 msgr2=0x562aeab6ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 shutdown_connections 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 1 -- 172.21.15.63:0/1853532978 wait complete. 2024-04-25T18:54:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.445+0000 7f2881975740 20 client.26197 trim_cache size 0 max 16384 2024-04-25T18:54:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 Processor -- start 2024-04-25T18:54:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 -- start start 2024-04-25T18:54:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6d2d0 0x562aeab94000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab94540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28781ae290 con 0x562aeab6d2d0 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28781abb00 con 0x562aeab6c9b0 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f2881975740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28781a9950 con 0x562aeab77b90 2024-04-25T18:54:45.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f287cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f287cc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56742/0 (socket says 172.21.15.63:56742) 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.462+0000 7f287cc00640 1 -- 172.21.15.63:0/3284788403 learned_addr learned my addr 172.21.15.63:0/3284788403 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab94540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f2877400640 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6d2d0 0x562aeab94000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 -- 172.21.15.63:0/3284788403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 msgr2=0x562aeab94540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.475 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab94540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 -- 172.21.15.63:0/3284788403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6d2d0 msgr2=0x562aeab94000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6d2d0 0x562aeab94000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 -- 172.21.15.63:0/3284788403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562aeab6d810 con 0x562aeab6c9b0 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f287cc00640 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x562aeab94a80 tx=0x7f287000ffb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f3bd5e6a5c378a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f2863e00640 1 -- 172.21.15.63:0/3284788403 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287000f9a0 con 0x562aeab6c9b0 2024-04-25T18:54:45.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f2863e00640 1 -- 172.21.15.63:0/3284788403 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287001f040 con 0x562aeab6c9b0 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.463+0000 7f2863e00640 1 -- 172.21.15.63:0/3284788403 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2870004fd0 con 0x562aeab6c9b0 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- 172.21.15.63:0/3284788403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 msgr2=0x562aeab87b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x562aeab94a80 tx=0x7f287000ffb0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- 172.21.15.63:0/3284788403 shutdown_connections 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeab94540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6d2d0 0x562aeab94000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 --2- 172.21.15.63:0/3284788403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6c9b0 0x562aeab87b40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- 172.21.15.63:0/3284788403 >> 172.21.15.63:0/3284788403 conn(0x562aeab68ec0 msgr2=0x562aeab73340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- 172.21.15.63:0/3284788403 shutdown_connections 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- 172.21.15.63:0/3284788403 wait complete. 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.464+0000 7f2881975740 1 -- start start 2024-04-25T18:54:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 Processor -- start 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeabea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28781a9fc0 con 0x562aeab6c9b0 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f28781ab430 con 0x562aeab6d2d0 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f28781aa690 con 0x562aeab77b90 2024-04-25T18:54:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2881975740 10 client.0 _ll_register_callbacks cb 0x7f2864009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2877400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f287cc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2877400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56758/0 (socket says 172.21.15.63:56758) 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.465+0000 7f2877400640 1 -- 172.21.15.63:0/2699910545 learned_addr learned my addr 172.21.15.63:0/2699910545 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f287cc00640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeabea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 -- 172.21.15.63:0/2699910545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 msgr2=0x562aeabea8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeabea8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 -- 172.21.15.63:0/2699910545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 msgr2=0x562aeab81310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab81310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 -- 172.21.15.63:0/2699910545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562aeabeaef0 con 0x562aeab6d2d0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2877400640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2864012700 tx=0x7f2864010780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b699645f472094 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2861600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864010c00 con 0x562aeab6d2d0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f287001f040 con 0x562aeab6d2d0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.466+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f2864010da0 con 0x562aeab6d2d0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f286400f920 con 0x562aeab6d2d0 2024-04-25T18:54:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 10 client.26177 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562aeabf22b0 con 0x562aeab6d2d0 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f287001f040 con 0x562aeab6d2d0 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f28640035f0 con 0x562aeab6d2d0 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 10 client.26177 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 20 client.26177 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeabf2970 0x562aeabf4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 4 client.26177 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:54:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.467+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562aeaae0320 con 0x562aeabf2970 2024-04-25T18:54:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.468+0000 7f287cc00640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeabf2970 0x562aeabf4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.468+0000 7f287cc00640 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeabf2970 0x562aeabf4d90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b143f8430848f0 server_cookie=7c9f774b2025e667 in_seq=0 out_seq=0 2024-04-25T18:54:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.468+0000 7f2861600640 10 client.26177 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:45.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 284787 4.17 MB/sec execute 284 sec latency 24.965 ms 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2861600640 1 -- 172.21.15.63:0/2699910545 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562aeaae0320 con 0x562aeabf2970 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2861600640 10 client.26177 handle_command_reply: tid=0 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 client.26177 shutdown 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 msgr2=0x562aeab81850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2864012700 tx=0x7f2864010780 comp rx=0 tx=0).stop 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 shutdown_connections 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562aeabf2970 0x562aeabf4d90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562aeab6d2d0 0x562aeab81850 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562aeab77b90 0x562aeabea8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 --2- 172.21.15.63:0/2699910545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562aeab6c9b0 0x562aeab81310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2861600640 0 client.26177 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 >> 172.21.15.63:0/2699910545 conn(0x562aeab68ec0 msgr2=0x562aeab68020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 shutdown_connections 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 1 -- 172.21.15.63:0/2699910545 wait complete. 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.528+0000 7f2881975740 20 client.26177 trim_cache size 0 max 16384 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:54:45.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 msgr2=0x7f285c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 0x7f285c0a6250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28680bcd80 tx=0x7f28680f48f0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 msgr2=0x7f28781985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2868005ab0 tx=0x7f2868003c10 comp rx=0 tx=0).stop 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 shutdown_connections 2024-04-25T18:54:45.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f285c0a3d90 0x7f285c0a6250 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f287819cea0 0x7f287819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2878107090 0x7f2878198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 --2- 172.21.15.63:0/738817587 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2878101fc0 0x7f28781985e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 >> 172.21.15.63:0/738817587 conn(0x7f28780fd8a0 msgr2=0x7f28780fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 shutdown_connections 2024-04-25T18:54:45.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:45.529+0000 7f287d600640 1 -- 172.21.15.63:0/738817587 wait complete. 2024-04-25T18:54:46.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 286120 4.18 MB/sec execute 285 sec latency 21.810 ms 2024-04-25T18:54:46.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:46 smithi114 ceph-mon[32988]: pgmap v254: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-25T18:54:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:46 smithi063 ceph-mon[28006]: pgmap v254: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-25T18:54:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:46 smithi070 ceph-mon[32820]: pgmap v254: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 207 op/s 2024-04-25T18:54:47.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 287585 4.20 MB/sec execute 286 sec latency 14.504 ms 2024-04-25T18:54:48.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 288825 4.21 MB/sec execute 287 sec latency 19.596 ms 2024-04-25T18:54:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:48 smithi114 ceph-mon[32988]: pgmap v255: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 979 KiB/s wr, 116 op/s 2024-04-25T18:54:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:48 smithi063 ceph-mon[28006]: pgmap v255: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 979 KiB/s wr, 116 op/s 2024-04-25T18:54:48.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:48 smithi070 ceph-mon[32820]: pgmap v255: 97 pgs: 97 active+clean; 92 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 979 KiB/s wr, 116 op/s 2024-04-25T18:54:49.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 290233 4.22 MB/sec execute 288 sec latency 21.271 ms 2024-04-25T18:54:50.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 291378 4.23 MB/sec execute 289 sec latency 14.163 ms 2024-04-25T18:54:50.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:50 smithi114 ceph-mon[32988]: pgmap v256: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-04-25T18:54:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:50 smithi063 ceph-mon[28006]: pgmap v256: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-04-25T18:54:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:50 smithi070 ceph-mon[32820]: pgmap v256: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.5 MiB/s wr, 227 op/s 2024-04-25T18:54:51.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 292323 4.23 MB/sec execute 290 sec latency 26.703 ms 2024-04-25T18:54:52.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 293428 4.23 MB/sec execute 291 sec latency 21.250 ms 2024-04-25T18:54:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:52 smithi114 ceph-mon[32988]: pgmap v257: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:52.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:52 smithi063 ceph-mon[28006]: pgmap v257: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:52 smithi070 ceph-mon[32820]: pgmap v257: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:53.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 294655 4.25 MB/sec execute 292 sec latency 22.237 ms 2024-04-25T18:54:54.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 295507 4.24 MB/sec execute 293 sec latency 35.247 ms 2024-04-25T18:54:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:54 smithi063 ceph-mon[28006]: pgmap v258: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:54 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:54 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:54 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:54 smithi070 ceph-mon[32820]: pgmap v258: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:54 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:54 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:54 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:54 smithi114 ceph-mon[32988]: pgmap v258: 97 pgs: 97 active+clean; 90 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-04-25T18:54:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:54 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:54:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:54 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:54 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:55.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 296990 4.25 MB/sec execute 294 sec latency 33.220 ms 2024-04-25T18:54:56.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:55 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:55 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:55 smithi114 ceph-mon[32988]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:54:56.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:55 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:55 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:55 smithi063 ceph-mon[28006]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:54:56.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:55 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:55 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:56.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:55 smithi070 ceph-mon[32820]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:54:56.217 INFO:tasks.quiescer.fs.[cephfs]:Total duration (49.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T18:54:56.217 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.8 2024-04-25T18:54:56.218 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:54:56.482 INFO:tasks.workunit.client.0.smithi063.stdout: 1 298344 4.27 MB/sec execute 295 sec latency 15.483 ms 2024-04-25T18:54:56.674 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:54:57.066 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:56 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:56 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:57.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:56 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:56 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:56 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:56 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:57.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:56 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:54:57.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:56 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:54:57.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:56 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:54:57.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 299383 4.27 MB/sec execute 296 sec latency 25.420 ms 2024-04-25T18:54:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:54:57 smithi114 ceph-mon[32988]: pgmap v260: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 162 op/s 2024-04-25T18:54:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:54:57 smithi063 ceph-mon[28006]: pgmap v260: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 162 op/s 2024-04-25T18:54:58.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:54:57 smithi070 ceph-mon[32820]: pgmap v260: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 162 op/s 2024-04-25T18:54:58.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 300392 4.27 MB/sec execute 297 sec latency 46.452 ms 2024-04-25T18:54:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 -- 172.21.15.63:0/302131620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f40742f0 msgr2=0x7f77f40746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 --2- 172.21.15.63:0/302131620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f40742f0 0x7f77f40746d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009900 tx=0x7f77e802a480 comp rx=0 tx=0).stop 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 -- 172.21.15.63:0/302131620 shutdown_connections 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 --2- 172.21.15.63:0/302131620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f4074c10 0x7f77f4112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 --2- 172.21.15.63:0/302131620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4112f40 0x7f77f4115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 --2- 172.21.15.63:0/302131620 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f40742f0 0x7f77f40746d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 -- 172.21.15.63:0/302131620 >> 172.21.15.63:0/302131620 conn(0x7f77f40fda00 msgr2=0x7f77f40ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 -- 172.21.15.63:0/302131620 shutdown_connections 2024-04-25T18:54:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.675+0000 7f77fc400640 1 -- 172.21.15.63:0/302131620 wait complete. 2024-04-25T18:54:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.676+0000 7f77fc400640 1 Processor -- start 2024-04-25T18:54:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.676+0000 7f77fc400640 1 -- start start 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f40742f0 0x7f77f41a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4074c10 0x7f77f41a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77f4118810 con 0x7f77f4074c10 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77f4118690 con 0x7f77f40742f0 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fc400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77f4118990 con 0x7f77f4112f40 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fa400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77f9000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4074c10 0x7f77f41a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77f9a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f40742f0 0x7f77f41a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fa400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45332/0 (socket says 172.21.15.63:45332) 2024-04-25T18:54:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.677+0000 7f77fa400640 1 -- 172.21.15.63:0/2834253273 learned_addr learned my addr 172.21.15.63:0/2834253273 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 -- 172.21.15.63:0/2834253273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f40742f0 msgr2=0x7f77f41a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f40742f0 0x7f77f41a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 -- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4074c10 msgr2=0x7f77f41a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4074c10 0x7f77f41a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 -- 172.21.15.63:0/2834253273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f419f2b0 con 0x7f77f4112f40 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77fa400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00eb40 tx=0x7f77ec00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.678+0000 7f77f2000640 1 -- 172.21.15.63:0/2834253273 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec004b20 con 0x7f77f4112f40 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.679+0000 7f77f2000640 1 -- 172.21.15.63:0/2834253273 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77ec004cc0 con 0x7f77f4112f40 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.679+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f419f5a0 con 0x7f77f4112f40 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.679+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f41ab7f0 con 0x7f77f4112f40 2024-04-25T18:54:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.679+0000 7f77f2000640 1 -- 172.21.15.63:0/2834253273 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec010720 con 0x7f77f4112f40 2024-04-25T18:54:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.683+0000 7f77ff18d740 1 Processor -- start 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.684+0000 7f77f2000640 1 -- 172.21.15.63:0/2834253273 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f77ec005560 con 0x7f77f4112f40 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.684+0000 7f77ff18d740 1 -- start start 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3af24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643c3a4d900 con 0x5643c3ae8140 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643c39f3ef0 con 0x5643c3af2a00 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77ff18d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643c3a498c0 con 0x5643c3ae7820 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77f9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77f9000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45334/0 (socket says 172.21.15.63:45334) 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77f9000640 1 -- 172.21.15.63:0/3227446135 learned_addr learned my addr 172.21.15.63:0/3227446135 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77f9a00640 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3af24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.687 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.685+0000 7f77f9000640 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 -- 172.21.15.63:0/3227446135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 msgr2=0x5643c3af4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 -- 172.21.15.63:0/3227446135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 msgr2=0x5643c3af24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3af24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 -- 172.21.15.63:0/3227446135 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643c3760210 con 0x5643c3ae7820 2024-04-25T18:54:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f9000640 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009fd0 tx=0x7f77e0005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a09845cc57ceac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77f2000640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 0x7f77dc0a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.686+0000 7f77e6a00640 1 -- 172.21.15.63:0/3227446135 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0004030 con 0x5643c3ae7820 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.687+0000 7f77e6a00640 1 -- 172.21.15.63:0/3227446135 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77e0005020 con 0x5643c3ae7820 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.687+0000 7f77f2000640 1 -- 172.21.15.63:0/2834253273 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f77ec0d0f00 con 0x7f77f4112f40 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.687+0000 7f77e6a00640 1 -- 172.21.15.63:0/3227446135 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0004a90 con 0x5643c3ae7820 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.687+0000 7f77f9a00640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 0x7f77dc0a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 -- 172.21.15.63:0/3227446135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 msgr2=0x5643c3ae7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77f9a00640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 0x7f77dc0a6280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f77e8005ab0 tx=0x7f77e8005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009fd0 tx=0x7f77e0005850 comp rx=0 tx=0).stop 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 -- 172.21.15.63:0/3227446135 shutdown_connections 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3af24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3227446135 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3ae7c00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 -- 172.21.15.63:0/3227446135 >> 172.21.15.63:0/3227446135 conn(0x5643c3ae3d30 msgr2=0x5643c3ae4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 -- 172.21.15.63:0/3227446135 shutdown_connections 2024-04-25T18:54:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.688+0000 7f77ff18d740 1 -- 172.21.15.63:0/3227446135 wait complete. 2024-04-25T18:54:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.689+0000 7f77ff18d740 1 -- start start 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.689+0000 7f77ff18d740 1 Processor -- start 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3aef970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643c3af5630 con 0x5643c3ae8140 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5643c3a07310 con 0x5643c3af2a00 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5643c3af57b0 con 0x5643c3ae7820 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77ff18d740 10 client.0 _ll_register_callbacks cb 0x5643c39d5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77f9000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77f9000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57412/0 (socket says 172.21.15.63:57412) 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77f9000640 1 -- 172.21.15.63:0/3466263140 learned_addr learned my addr 172.21.15.63:0/3466263140 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.690+0000 7f77fa400640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77f9a00640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3aef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 -- 172.21.15.63:0/3466263140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 msgr2=0x5643c3aef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3aef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 -- 172.21.15.63:0/3466263140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 msgr2=0x5643c3aefeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 -- 172.21.15.63:0/3466263140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643c382c620 con 0x5643c3af2a00 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77f9000640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77f9a00640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3aef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77fa400640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af03f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0ce5e0 tx=0x7f77ec09c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1638e7e80b1f1982 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.691+0000 7f77db400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:54:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.692+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0b5020 con 0x5643c3af2a00 2024-04-25T18:54:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.692+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f77ec09d0c0 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.692+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77e0005020 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.692+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f77ec0a5ad0 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 10 client.26189 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5643c3b018d0 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77e0005020 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f77ec098bb0 con 0x5643c3af2a00 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 10 client.26189 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 10 client.26189 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 20 client.26189 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b02d90 0x5643c3b051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 4 client.26189 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.693+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643c3a618b0 con 0x5643c3b02d90 2024-04-25T18:54:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.694+0000 7f77f9a00640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b02d90 0x5643c3b051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.694+0000 7f77f9a00640 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b02d90 0x5643c3b051a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1901019fbe1fb38d server_cookie=ae40f6da1d1ba8f2 in_seq=0 out_seq=0 2024-04-25T18:54:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.694+0000 7f77db400640 10 client.26189 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.695+0000 7f77db400640 1 -- 172.21.15.63:0/3466263140 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643c3a618b0 con 0x5643c3b02d90 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.695+0000 7f77db400640 10 client.26189 handle_command_reply: tid=0 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.695+0000 7f77ff18d740 1 client.26189 shutdown 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 msgr2=0x5643c3af03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af03f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0ce5e0 tx=0x7f77ec09c880 comp rx=0 tx=0).stop 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 shutdown_connections 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b02d90 0x5643c3b051a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af03f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3aefeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77db400640 0 client.26189 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 --2- 172.21.15.63:0/3466263140 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3aef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 >> 172.21.15.63:0/3466263140 conn(0x5643c3ae3d30 msgr2=0x5643c3ae9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 shutdown_connections 2024-04-25T18:54:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 1 -- 172.21.15.63:0/3466263140 wait complete. 2024-04-25T18:54:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.696+0000 7f77ff18d740 20 client.26189 trim_cache size 0 max 16384 2024-04-25T18:54:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.719+0000 7f77ff18d740 1 Processor -- start 2024-04-25T18:54:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.719+0000 7f77ff18d740 1 -- start start 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.719+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3afe4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77ff18d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b6b50 con 0x5643c3ae8140 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77ff18d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b43c0 con 0x5643c3af2a00 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77ff18d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b2230 con 0x5643c3ae7820 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45344/0 (socket says 172.21.15.63:45344) 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9a00640 1 -- 172.21.15.63:0/990193908 learned_addr learned my addr 172.21.15.63:0/990193908 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9000640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3afe4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 -- 172.21.15.63:0/990193908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 msgr2=0x5643c3b05180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 -- 172.21.15.63:0/990193908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 msgr2=0x5643c3afe4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3afe4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77fa400640 1 -- 172.21.15.63:0/990193908 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643c3ae7f70 con 0x5643c3af2a00 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9a00640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.720+0000 7f77f9000640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3afe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.721+0000 7f77fa400640 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af0210 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0cec10 tx=0x7f77ec09c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c868675a5b144c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.721+0000 7f77e6000640 1 -- 172.21.15.63:0/990193908 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec083070 con 0x5643c3af2a00 2024-04-25T18:54:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.721+0000 7f77e6000640 1 -- 172.21.15.63:0/990193908 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77ec09d0c0 con 0x5643c3af2a00 2024-04-25T18:54:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.721+0000 7f77e6000640 1 -- 172.21.15.63:0/990193908 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0b5070 con 0x5643c3af2a00 2024-04-25T18:54:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- 172.21.15.63:0/990193908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 msgr2=0x5643c3af0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af0210 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0cec10 tx=0x7f77ec09c8f0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- 172.21.15.63:0/990193908 shutdown_connections 2024-04-25T18:54:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3af2a00 0x5643c3af0210 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3ae8140 0x5643c3afe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 --2- 172.21.15.63:0/990193908 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3b05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- 172.21.15.63:0/990193908 >> 172.21.15.63:0/990193908 conn(0x5643c3ae3d30 msgr2=0x5643c3ae2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- 172.21.15.63:0/990193908 shutdown_connections 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- 172.21.15.63:0/990193908 wait complete. 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.722+0000 7f77ff18d740 1 -- start start 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 Processor -- start 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 0x5643c3af8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3af2a00 0x5643c3af8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b2880 con 0x5643c3af2a00 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b3cf0 con 0x5643c3ae8140 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77f41b2f50 con 0x5643c3ae7820 2024-04-25T18:54:58.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77ff18d740 10 client.0 _ll_register_callbacks cb 0x7f77ec09c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77f9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77f9a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45348/0 (socket says 172.21.15.63:45348) 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77f9a00640 1 -- 172.21.15.63:0/716928542 learned_addr learned my addr 172.21.15.63:0/716928542 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77f9000640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 0x5643c3af8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.723+0000 7f77fa400640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3af2a00 0x5643c3af8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 -- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 msgr2=0x5643c3af8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 0x5643c3af8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 -- 172.21.15.63:0/716928542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3af2a00 msgr2=0x5643c3af8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3af2a00 0x5643c3af8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 -- 172.21.15.63:0/716928542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643c3af9010 con 0x5643c3ae7820 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9000640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 0x5643c3af8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77f9a00640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77e802a9a0 tx=0x7f77e8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e592bd8673a6e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77db400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e8035040 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77ec09d0c0 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.724+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f77e8004b80 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f77e8004d80 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77ff18d740 10 client.26221 fetch_fsmap learned FSMap version 34 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5643c3b69470 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77ec09d0c0 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f77e8003710 con 0x5643c3ae7820 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77ff18d740 10 client.26221 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.725+0000 7f77ff18d740 10 client.26221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.726+0000 7f77ff18d740 20 client.26221 populate_metadata read hostname 'smithi063' 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.726+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b69ba0 0x5643c3b6bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:58.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.726+0000 7f77ff18d740 4 client.26221 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:54:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.726+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643c3a5b320 con 0x5643c3b69ba0 2024-04-25T18:54:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.726+0000 7f77f9000640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b69ba0 0x5643c3b6bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.727+0000 7f77f9000640 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b69ba0 0x5643c3b6bfc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9899493a8dff44 server_cookie=574b2f82350ce35d in_seq=0 out_seq=0 2024-04-25T18:54:58.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.727+0000 7f77db400640 10 client.26221 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77db400640 1 -- 172.21.15.63:0/716928542 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643c3a5b320 con 0x5643c3b69ba0 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77db400640 10 client.26221 handle_command_reply: tid=0 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 client.26221 shutdown 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 msgr2=0x5643c3afaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f77e802a9a0 tx=0x7f77e8009260 comp rx=0 tx=0).stop 2024-04-25T18:54:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 shutdown_connections 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5643c3b69ba0 0x5643c3b6bfc0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5643c3ae8140 0x5643c3af8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643c3af2a00 0x5643c3af8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 --2- 172.21.15.63:0/716928542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5643c3ae7820 0x5643c3afaea0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.774+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 >> 172.21.15.63:0/716928542 conn(0x5643c3ae3d30 msgr2=0x5643c3ae2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.775+0000 7f77db400640 0 client.26221 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.775+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 shutdown_connections 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.775+0000 7f77ff18d740 1 -- 172.21.15.63:0/716928542 wait complete. 2024-04-25T18:54:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.775+0000 7f77ff18d740 20 client.26221 trim_cache size 0 max 16384 2024-04-25T18:54:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:54:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:54:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:54:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.776+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 msgr2=0x7f77dc0a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.776+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 0x7f77dc0a6280 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f77e8005ab0 tx=0x7f77e8005a20 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.776+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 msgr2=0x7f77f419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.776+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00eb40 tx=0x7f77ec00c2d0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.776+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 shutdown_connections 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f77dc0a3dc0 0x7f77dc0a6280 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f77f4112f40 0x7f77f419eb30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77f4074c10 0x7f77f41a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 --2- 172.21.15.63:0/2834253273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77f40742f0 0x7f77f41a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 >> 172.21.15.63:0/2834253273 conn(0x7f77f40fda00 msgr2=0x7f77f40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 shutdown_connections 2024-04-25T18:54:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:54:58.777+0000 7f77fc400640 1 -- 172.21.15.63:0/2834253273 wait complete. 2024-04-25T18:54:59.424 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.422+0000 7f1138a00700 1 -- 172.21.15.63:0/1832663409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134105860 msgr2=0x7f11341098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.422+0000 7f1138a00700 1 --2- 172.21.15.63:0/1832663409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134105860 0x7f11341098b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f11200099c0 tx=0x7f1120009cd0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 -- 172.21.15.63:0/1832663409 shutdown_connections 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 --2- 172.21.15.63:0/1832663409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f1134111b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 --2- 172.21.15.63:0/1832663409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134104eb0 0x7f1134105290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 --2- 172.21.15.63:0/1832663409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134105860 0x7f11341098b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 -- 172.21.15.63:0/1832663409 >> 172.21.15.63:0/1832663409 conn(0x7f1134100780 msgr2=0x7f1134102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:59.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 -- 172.21.15.63:0/1832663409 shutdown_connections 2024-04-25T18:54:59.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.423+0000 7f1138a00700 1 -- 172.21.15.63:0/1832663409 wait complete. 2024-04-25T18:54:59.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.424+0000 7f1138a00700 1 Processor -- start 2024-04-25T18:54:59.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.424+0000 7f1138a00700 1 -- start start 2024-04-25T18:54:59.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134105860 0x7f113419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f113419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1134114e00 con 0x7f1134109f90 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1134114c80 con 0x7f1134105860 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1138a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1134114f80 con 0x7f1134104eb0 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1131e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1131e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45366/0 (socket says 172.21.15.63:45366) 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1131e00700 1 -- 172.21.15.63:0/640837318 learned_addr learned my addr 172.21.15.63:0/640837318 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1131400700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134105860 0x7f113419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.425+0000 7f1132800700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f113419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 -- 172.21.15.63:0/640837318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134105860 msgr2=0x7f113419b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134105860 0x7f113419b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 -- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 msgr2=0x7f113419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:59.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f113419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 -- 172.21.15.63:0/640837318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11341a0330 con 0x7f1134104eb0 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1132800700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f113419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1131e00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f112400eb30 tx=0x7f112400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112400cc90 con 0x7f1134104eb0 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11240044d0 con 0x7f1134104eb0 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1124010430 con 0x7f1134104eb0 2024-04-25T18:54:59.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.426+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11341a0620 con 0x7f1134104eb0 2024-04-25T18:54:59.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.427+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11341a7ea0 con 0x7f1134104eb0 2024-04-25T18:54:59.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.428+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1134106390 con 0x7f1134104eb0 2024-04-25T18:54:59.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.430+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1124004020 con 0x7f1134104eb0 2024-04-25T18:54:59.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.432+0000 7f112aa00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 0x7f11180a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:54:59.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.432+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f11240d0ac0 con 0x7f1134104eb0 2024-04-25T18:54:59.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.432+0000 7f1131400700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 0x7f11180a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:54:59.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.433+0000 7f1131400700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 0x7f11180a62f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11200098e0 tx=0x7f1120005960 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:54:59.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.436+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f112409d0c0 con 0x7f1134104eb0 2024-04-25T18:54:59.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 301748 4.29 MB/sec execute 298 sec latency 26.366 ms 2024-04-25T18:54:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.788+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f113419c6b0 con 0x7f1134104eb0 2024-04-25T18:54:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.789+0000 7f112aa00700 1 -- 172.21.15.63:0/640837318 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11871 (secure 0 0 0) 0x7f113419c6b0 con 0x7f1134104eb0 2024-04-25T18:54:59.790 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:54:59.790 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:53:53.997852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:54:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 msgr2=0x7f11180a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 0x7f11180a62f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f11200098e0 tx=0x7f1120005960 comp rx=0 tx=0).stop 2024-04-25T18:54:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 msgr2=0x7f113419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:54:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f112400eb30 tx=0x7f112400ee40 comp rx=0 tx=0).stop 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 shutdown_connections 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11180a3e30 0x7f11180a62f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1134104eb0 0x7f113419b370 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1134109f90 0x7f113419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 --2- 172.21.15.63:0/640837318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1134105860 0x7f113419b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 >> 172.21.15.63:0/640837318 conn(0x7f1134100780 msgr2=0x7f1134101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 shutdown_connections 2024-04-25T18:54:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:54:59.792+0000 7f1138a00700 1 -- 172.21.15.63:0/640837318 wait complete. 2024-04-25T18:54:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 34 2024-04-25T18:55:00.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:00 smithi070 ceph-mon[32820]: pgmap v261: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 255 op/s 2024-04-25T18:55:00.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/640837318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:00.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 302630 4.29 MB/sec execute 299 sec latency 39.013 ms 2024-04-25T18:55:00.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:00 smithi114 ceph-mon[32988]: pgmap v261: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 255 op/s 2024-04-25T18:55:00.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:00 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/640837318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:00 smithi063 ceph-mon[28006]: pgmap v261: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.8 MiB/s wr, 255 op/s 2024-04-25T18:55:00.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:00 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/640837318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:00.776 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 84.8 --await-for 42.0' 2024-04-25T18:55:00.777 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 84.8 --await-for 42.0 2024-04-25T18:55:01.283 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:55:01.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 303811 4.29 MB/sec execute 300 sec latency 31.996 ms 2024-04-25T18:55:02.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:02 smithi063 ceph-mon[28006]: pgmap v262: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:02.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:02 smithi070 ceph-mon[32820]: pgmap v262: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:02.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 305137 4.31 MB/sec execute 301 sec latency 30.676 ms 2024-04-25T18:55:02.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:02 smithi114 ceph-mon[32988]: pgmap v262: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:03.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306002 4.30 MB/sec execute 302 sec latency 35.449 ms 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.920+0000 7f17fd800700 1 -- 172.21.15.63:0/633121645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f8102470 msgr2=0x7f17f8102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.920+0000 7f17fd800700 1 --2- 172.21.15.63:0/633121645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f8102470 0x7f17f8102850 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009990 tx=0x7f17e8009ca0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 -- 172.21.15.63:0/633121645 shutdown_connections 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 --2- 172.21.15.63:0/633121645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f8102470 0x7f17f8102850 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 --2- 172.21.15.63:0/633121645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f810f4f0 0x7f17f81118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 --2- 172.21.15.63:0/633121645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102d90 0x7f17f810efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 -- 172.21.15.63:0/633121645 >> 172.21.15.63:0/633121645 conn(0x7f17f80788e0 msgr2=0x7f17f8078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 -- 172.21.15.63:0/633121645 shutdown_connections 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.921+0000 7f17fd800700 1 -- 172.21.15.63:0/633121645 wait complete. 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.922+0000 7f17fd800700 1 Processor -- start 2024-04-25T18:55:03.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.922+0000 7f17fd800700 1 -- start start 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102470 0x7f17f819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 0x7f17f819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8114e90 con 0x7f17f8102470 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8114d10 con 0x7f17f8102d90 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17fd800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17f8115010 con 0x7f17f810f4f0 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.923+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102470 0x7f17f819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45394/0 (socket says 172.21.15.63:45394) 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 -- 172.21.15.63:0/3872490965 learned_addr learned my addr 172.21.15.63:0/3872490965 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f6000700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 0x7f17f819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 -- 172.21.15.63:0/3872490965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 msgr2=0x7f17f819b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 0x7f17f819b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 -- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102470 msgr2=0x7f17f819b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102470 0x7f17f819b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 -- 172.21.15.63:0/3872490965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f81a0380 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f6000700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 0x7f17f819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.924+0000 7f17f7400700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17f000ef90 tx=0x7f17f000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.925+0000 7f17ef400700 1 -- 172.21.15.63:0/3872490965 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f00092e0 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.925+0000 7f17ef400700 1 -- 172.21.15.63:0/3872490965 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17f0004bb0 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.925+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f81a0610 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.925+0000 7f17ef400700 1 -- 172.21.15.63:0/3872490965 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0004280 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.925+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81a7ef0 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.928+0000 7f17ef400700 1 -- 172.21.15.63:0/3872490965 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f17f001b030 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.930+0000 7f17ef400700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 0x7f17dc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.930+0000 7f17ef400700 1 -- 172.21.15.63:0/3872490965 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f17f00d0290 con 0x7f17f810f4f0 2024-04-25T18:55:03.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.930+0000 7f17f6a00700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 0x7f17dc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.931+0000 7f1808253b80 1 Processor -- start 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.931+0000 7f1808253b80 1 -- start start 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.931+0000 7f17f6a00700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 0x7f17dc0a6390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b2e0 tx=0x7f17e8005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.931+0000 7f1808253b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x5567741831b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f1808253b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f1808253b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774199160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f1808253b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5567740d6020 con 0x556774196d70 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f1808253b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55677407c8c0 con 0x5567741836f0 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f1808253b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5567740d1fe0 con 0x556774182dd0 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57402/0 (socket says 172.21.15.63:57402) 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6000700 1 -- 172.21.15.63:0/1727461221 learned_addr learned my addr 172.21.15.63:0/1727461221 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:03.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f7400700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6a00700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x5567741831b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6000700 1 -- 172.21.15.63:0/1727461221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 msgr2=0x5567741831b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.932+0000 7f17f6000700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x5567741831b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17f6000700 1 -- 172.21.15.63:0/1727461221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 msgr2=0x556774199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17f6000700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17f6000700 1 -- 172.21.15.63:0/1727461221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556774072810 con 0x5567741836f0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17f6a00700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x5567741831b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17f6000700 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f17e4011540 tx=0x7f17e4011850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c434d101be416ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17e3e00700 1 -- 172.21.15.63:0/1727461221 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004a30 con 0x5567741836f0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17e3e00700 1 -- 172.21.15.63:0/1727461221 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e40054d0 con 0x5567741836f0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.933+0000 7f17e3e00700 1 -- 172.21.15.63:0/1727461221 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e400e870 con 0x5567741836f0 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 -- 172.21.15.63:0/1727461221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 msgr2=0x556774196830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f17e4011540 tx=0x7f17e4011850 comp rx=0 tx=0).stop 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 -- 172.21.15.63:0/1727461221 shutdown_connections 2024-04-25T18:55:03.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x5567741831b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 --2- 172.21.15.63:0/1727461221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5567741836f0 0x556774196830 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 -- 172.21.15.63:0/1727461221 >> 172.21.15.63:0/1727461221 conn(0x5567740b20e0 msgr2=0x55677417f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 -- 172.21.15.63:0/1727461221 shutdown_connections 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.934+0000 7f1808253b80 1 -- 172.21.15.63:0/1727461221 wait complete. 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.935+0000 7f1808253b80 1 -- start start 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.935+0000 7f1808253b80 1 Processor -- start 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567741836f0 0x556774193780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774193cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556774199740 con 0x556774196d70 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55677408f9f0 con 0x556774182dd0 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5567741998c0 con 0x5567741836f0 2024-04-25T18:55:03.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f1808253b80 10 client.0 _ll_register_callbacks cb 0x55677406e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57406/0 (socket says 172.21.15.63:57406) 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f17f6a00700 1 -- 172.21.15.63:0/1921259111 learned_addr learned my addr 172.21.15.63:0/1921259111 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f17f6000700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567741836f0 0x556774193780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.936+0000 7f17f7400700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774193cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 -- 172.21.15.63:0/1921259111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567741836f0 msgr2=0x556774193780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567741836f0 0x556774193780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 -- 172.21.15.63:0/1921259111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 msgr2=0x556774193cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774193cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 -- 172.21.15.63:0/1921259111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556774023a60 con 0x556774182dd0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17f6a00700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b950 tx=0x7f17e8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=647855901e931aed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:03.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.937+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8003890 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e40054d0 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f17e8003a30 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f17e8003c30 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f1808253b80 10 client.26213 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55677406cc80 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e40054d0 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.938+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f17e800f460 con 0x556774182dd0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 10 client.26213 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 10 client.26213 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 20 client.26213 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5567741a6450 0x5567741a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 4 client.26213 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567740e3810 con 0x5567741a6450 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.939+0000 7f17f6000700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5567741a6450 0x5567741a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.940+0000 7f17e0c00700 10 client.26213 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.940+0000 7f17f6000700 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5567741a6450 0x5567741a8840 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14caeb995937fdda server_cookie=2c5574520b3a467a in_seq=0 out_seq=0 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f17e0c00700 1 -- 172.21.15.63:0/1921259111 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5567740e3810 con 0x5567741a6450 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f17e0c00700 10 client.26213 handle_command_reply: tid=0 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 client.26213 shutdown 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 msgr2=0x556774196480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b950 tx=0x7f17e8009310 comp rx=0 tx=0).stop 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 shutdown_connections 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5567741a6450 0x5567741a8840 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556774196d70 0x556774193cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x556774196480 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 --2- 172.21.15.63:0/1921259111 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5567741836f0 0x556774193780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f17e0c00700 0 client.26213 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 >> 172.21.15.63:0/1921259111 conn(0x5567740b20e0 msgr2=0x55677417eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:03.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 shutdown_connections 2024-04-25T18:55:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.941+0000 7f1808253b80 1 -- 172.21.15.63:0/1921259111 wait complete. 2024-04-25T18:55:03.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.942+0000 7f1808253b80 20 client.26213 trim_cache size 0 max 16384 2024-04-25T18:55:03.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.967+0000 7f1808253b80 1 Processor -- start 2024-04-25T18:55:03.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.967+0000 7f1808253b80 1 -- start start 2024-04-25T18:55:03.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x556774195840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774196d70 0x5567741ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81adae0 con 0x5567741836f0 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ab310 con 0x556774182dd0 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f1808253b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81a92d0 con 0x556774196d70 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f17f7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774196d70 0x5567741ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57408/0 (socket says 172.21.15.63:57408) 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f17f6a00700 1 -- 172.21.15.63:0/2787564454 learned_addr learned my addr 172.21.15.63:0/2787564454 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.968+0000 7f17f6000700 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x556774195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 -- 172.21.15.63:0/2787564454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774196d70 msgr2=0x5567741ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774196d70 0x5567741ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 -- 172.21.15.63:0/2787564454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 msgr2=0x556774195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x556774195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 -- 172.21.15.63:0/2787564454 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567741a8ea0 con 0x556774182dd0 2024-04-25T18:55:03.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6000700 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x556774195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:55:03.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009ce0 tx=0x7f17e800b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14504f8977eab0bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17e3400700 1 -- 172.21.15.63:0/2787564454 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e800be80 con 0x556774182dd0 2024-04-25T18:55:03.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17e3400700 1 -- 172.21.15.63:0/2787564454 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e8003890 con 0x556774182dd0 2024-04-25T18:55:03.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.969+0000 7f17e3400700 1 -- 172.21.15.63:0/2787564454 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8039930 con 0x556774182dd0 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 -- 172.21.15.63:0/2787564454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 msgr2=0x5567741a26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009ce0 tx=0x7f17e800b390 comp rx=0 tx=0).stop 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 -- 172.21.15.63:0/2787564454 shutdown_connections 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774196d70 0x5567741ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774182dd0 0x5567741a26a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 --2- 172.21.15.63:0/2787564454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x556774195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 -- 172.21.15.63:0/2787564454 >> 172.21.15.63:0/2787564454 conn(0x5567740b20e0 msgr2=0x55677418ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 -- 172.21.15.63:0/2787564454 shutdown_connections 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.970+0000 7f1808253b80 1 -- 172.21.15.63:0/2787564454 wait complete. 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 -- start start 2024-04-25T18:55:03.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 Processor -- start 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x55677419c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 0x55677419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81a97d0 con 0x5567741836f0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81aac40 con 0x556774196d70 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81a9ea0 con 0x556774182dd0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.971+0000 7f1808253b80 10 client.0 _ll_register_callbacks cb 0x55677406e8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45412/0 (socket says 172.21.15.63:45412) 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f6a00700 1 -- 172.21.15.63:0/2442281435 learned_addr learned my addr 172.21.15.63:0/2442281435 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 0x55677419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f6000700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x55677419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 -- 172.21.15.63:0/2442281435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 msgr2=0x55677419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 -- 172.21.15.63:0/2442281435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 msgr2=0x55677419c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x55677419c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 -- 172.21.15.63:0/2442281435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567741847e0 con 0x556774196d70 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17f7400700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 0x55677419c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f17f00cd940 tx=0x7f17f00cdea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d15035528e552a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.972+0000 7f17e0c00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f00a4b70 con 0x556774196d70 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 8587+0+0 (secure 0 0 0) 0x7f17f00a00c0 con 0x556774196d70 2024-04-25T18:55:03.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e8003890 con 0x556774196d70 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f17f0013ea0 con 0x556774196d70 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f1808253b80 10 client.26219 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x556774183d70 con 0x556774196d70 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e8003890 con 0x556774196d70 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.973+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f17f0097f70 con 0x556774196d70 2024-04-25T18:55:03.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 10 client.26219 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 10 client.26219 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 20 client.26219 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55677420ea70 0x556774210e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 4 client.26219 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.8, "await_for": 42.0}] 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.8, "await_for": 42.0}) -- 0x5567740dd8b0 con 0x55677420ea70 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55677420ea70 0x556774210e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f17f6a00700 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55677420ea70 0x556774210e90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc284fad046f5d33 server_cookie=276c13a2b34a8035 in_seq=0 out_seq=0 2024-04-25T18:55:03.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:03.974+0000 7f17e0c00700 10 client.26219 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f17e0c00700 1 -- 172.21.15.63:0/2442281435 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5567740dd8b0 con 0x55677420ea70 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f17e0c00700 10 client.26219 handle_command_reply: tid=0 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 client.26219 shutdown 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 msgr2=0x55677419c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 0x55677419c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f17f00cd940 tx=0x7f17f00cdea0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 shutdown_connections 2024-04-25T18:55:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55677420ea70 0x556774210e90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x556774182dd0 0x55677419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556774196d70 0x55677419c6a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 --2- 172.21.15.63:0/2442281435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5567741836f0 0x55677419c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 >> 172.21.15.63:0/2442281435 conn(0x5567740b20e0 msgr2=0x556774195840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.027+0000 7f17e0c00700 0 client.26219 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.028+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 shutdown_connections 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.028+0000 7f1808253b80 1 -- 172.21.15.63:0/2442281435 wait complete. 2024-04-25T18:55:04.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.028+0000 7f1808253b80 20 client.26219 trim_cache size 0 max 16384 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 34, 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 26, 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "2a159cfd": { 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 26, 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:04.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 84.8, 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:04.032 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 msgr2=0x7f17dc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 0x7f17dc0a6390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17e800b2e0 tx=0x7f17e8005fd0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 msgr2=0x7f17f819fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17f000ef90 tx=0x7f17f000c3a0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 shutdown_connections 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f17dc0a3ed0 0x7f17dc0a6390 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f17f810f4f0 0x7f17f819fc40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f17f8102470 0x7f17f819b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 --2- 172.21.15.63:0/3872490965 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f17f8102d90 0x7f17f819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:04.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.029+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 >> 172.21.15.63:0/3872490965 conn(0x7f17f80788e0 msgr2=0x7f17f81109e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:04.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.030+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 shutdown_connections 2024-04-25T18:55:04.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:04.030+0000 7f17fd800700 1 -- 172.21.15.63:0/3872490965 wait complete. 2024-04-25T18:55:04.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:04 smithi070 ceph-mon[32820]: pgmap v263: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:04.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.30 MB/sec execute 303 sec latency 453.906 ms 2024-04-25T18:55:04.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:04 smithi114 ceph-mon[32988]: pgmap v263: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:04.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:04 smithi063 ceph-mon[28006]: pgmap v263: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 144 op/s 2024-04-25T18:55:04.886 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2a159cfd', quiesce took 8.7 seconds. Will release after: 24.8 2024-04-25T18:55:05.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.29 MB/sec execute 304 sec latency 1454.060 ms 2024-04-25T18:55:05.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:05 smithi114 ceph-mon[32988]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T18:55:05.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:05 smithi063 ceph-mon[28006]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T18:55:05.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:05 smithi070 ceph-mon[32820]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T18:55:06.483 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.27 MB/sec execute 305 sec latency 2454.160 ms 2024-04-25T18:55:06.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:06 smithi114 ceph-mon[32988]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:06.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:06 smithi063 ceph-mon[28006]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:06 smithi070 ceph-mon[32820]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:07.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.26 MB/sec execute 306 sec latency 3454.317 ms 2024-04-25T18:55:08.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.25 MB/sec execute 307 sec latency 4454.477 ms 2024-04-25T18:55:08.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:08 smithi114 ceph-mon[32988]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:08.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:08 smithi063 ceph-mon[28006]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:08 smithi070 ceph-mon[32820]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 240 op/s 2024-04-25T18:55:09.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.23 MB/sec execute 308 sec latency 5454.629 ms 2024-04-25T18:55:10.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.22 MB/sec execute 309 sec latency 6454.778 ms 2024-04-25T18:55:10.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:10 smithi114 ceph-mon[32988]: pgmap v267: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:10 smithi063 ceph-mon[28006]: pgmap v267: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:10 smithi070 ceph-mon[32820]: pgmap v267: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:11.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.20 MB/sec execute 310 sec latency 7454.871 ms 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 -- 172.21.15.63:0/549399271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 msgr2=0x7f82c01128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 --2- 172.21.15.63:0/549399271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c01128a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009900 tx=0x7f82ac02a3f0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 -- 172.21.15.63:0/549399271 shutdown_connections 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 --2- 172.21.15.63:0/549399271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 0x7f82c01151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 --2- 172.21.15.63:0/549399271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c01128a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 --2- 172.21.15.63:0/549399271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c0076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 -- 172.21.15.63:0/549399271 >> 172.21.15.63:0/549399271 conn(0x7f82c00ffc50 msgr2=0x7f82c0102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 -- 172.21.15.63:0/549399271 shutdown_connections 2024-04-25T18:55:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.925+0000 7f82c5400640 1 -- 172.21.15.63:0/549399271 wait complete. 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.926+0000 7f82c5400640 1 Processor -- start 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 -- start start 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 0x7f82c01a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0118800 con 0x7f82c0076da0 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0118680 con 0x7f82c0112de0 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82c5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82c0118980 con 0x7f82c0076480 2024-04-25T18:55:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82bde00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82bde00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c019f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33024/0 (socket says 172.21.15.63:33024) 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82bde00640 1 -- 172.21.15.63:0/858230597 learned_addr learned my addr 172.21.15.63:0/858230597 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.927+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82bf200640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 0x7f82c01a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 -- 172.21.15.63:0/858230597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 msgr2=0x7f82c01a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 0x7f82c01a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 -- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 msgr2=0x7f82c019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 -- 172.21.15.63:0/858230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c01a3d20 con 0x7f82c0076480 2024-04-25T18:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.928+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c019ed80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f82b000e9f0 tx=0x7f82b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.929+0000 7f82b7400640 1 -- 172.21.15.63:0/858230597 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b000ccc0 con 0x7f82c0076480 2024-04-25T18:55:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.929+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c01a4010 con 0x7f82c0076480 2024-04-25T18:55:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.929+0000 7f82b7400640 1 -- 172.21.15.63:0/858230597 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b0005020 con 0x7f82c0076480 2024-04-25T18:55:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.929+0000 7f82b7400640 1 -- 172.21.15.63:0/858230597 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b0004a00 con 0x7f82c0076480 2024-04-25T18:55:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.929+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c01ab8a0 con 0x7f82c0076480 2024-04-25T18:55:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.932+0000 7f82c8218740 1 Processor -- start 2024-04-25T18:55:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.932+0000 7f82c8218740 1 -- start start 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82b7400640 1 -- 172.21.15.63:0/858230597 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f82b0004ba0 con 0x7f82c0076480 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7241140 0x5577a724b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 0x5577a724ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5577a71a6900 con 0x5577a7240820 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5577a714cef0 con 0x5577a724ba00 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82c8218740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5577a71a28c0 con 0x5577a7241140 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33036/0 (socket says 172.21.15.63:33036) 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82bf200640 1 -- 172.21.15.63:0/3627025706 learned_addr learned my addr 172.21.15.63:0/3627025706 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82bde00640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 0x5577a724ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82be800640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7241140 0x5577a724b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.933+0000 7f82bf200640 1 -- 172.21.15.63:0/3627025706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7241140 msgr2=0x5577a724b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bf200640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7241140 0x5577a724b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bf200640 1 -- 172.21.15.63:0/3627025706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 msgr2=0x5577a724ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bf200640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 0x5577a724ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bf200640 1 -- 172.21.15.63:0/3627025706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a6eb9210 con 0x5577a7240820 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bde00640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 0x5577a724ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82bf200640 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009f30 tx=0x7f82b80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fd06538d61d8ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82ab400640 1 -- 172.21.15.63:0/3627025706 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8004d20 con 0x5577a7240820 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82ab400640 1 -- 172.21.15.63:0/3627025706 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b8004050 con 0x5577a7240820 2024-04-25T18:55:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82ab400640 1 -- 172.21.15.63:0/3627025706 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8005220 con 0x5577a7240820 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.934+0000 7f82b7400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 0x7f82a40a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82b7400640 1 -- 172.21.15.63:0/858230597 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f82b00d0800 con 0x7f82c0076480 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 0x7f82a40a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 -- 172.21.15.63:0/3627025706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 msgr2=0x5577a7240c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009f30 tx=0x7f82b80118d0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 -- 172.21.15.63:0/3627025706 shutdown_connections 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a724ba00 0x5577a724ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7241140 0x5577a724b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 --2- 172.21.15.63:0/3627025706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7240c00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 -- 172.21.15.63:0/3627025706 >> 172.21.15.63:0/3627025706 conn(0x5577a723cd30 msgr2=0x5577a723d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 -- 172.21.15.63:0/3627025706 shutdown_connections 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82c8218740 1 -- 172.21.15.63:0/3627025706 wait complete. 2024-04-25T18:55:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.935+0000 7f82be800640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 0x7f82a40a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82b00cdf90 tx=0x7f82b00d6550 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.936+0000 7f82c8218740 1 -- start start 2024-04-25T18:55:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.936+0000 7f82c8218740 1 Processor -- start 2024-04-25T18:55:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.936+0000 7f82c8218740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a72493e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5577a724e630 con 0x5577a7240820 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5577a7160310 con 0x5577a7241140 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5577a724e7b0 con 0x5577a724ba00 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82c8218740 10 client.0 _ll_register_callbacks cb 0x5577a712ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33044/0 (socket says 172.21.15.63:33044) 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 -- 172.21.15.63:0/219285370 learned_addr learned my addr 172.21.15.63:0/219285370 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bde00640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a72493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 -- 172.21.15.63:0/219285370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 msgr2=0x5577a72493e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a72493e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 -- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 msgr2=0x5577a7248ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.937+0000 7f82bf200640 1 -- 172.21.15.63:0/219285370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a6f85620 con 0x5577a7240820 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f82bf200640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f82b8004e70 tx=0x7f82b8009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d385a3ce87776ea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f829fe00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8004400 con 0x5577a7240820 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f82b80045a0 con 0x5577a7240820 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82b8004050 con 0x5577a7240820 2024-04-25T18:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f82b8036050 con 0x5577a7240820 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f82c8218740 10 client.16050 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.938+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5577a725a850 con 0x5577a7240820 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b8004050 con 0x5577a7240820 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f82b8028480 con 0x5577a7240820 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 10 client.16050 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 20 client.16050 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a725bc50 0x5577a725e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 4 client.16050 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577a71ba8b0 con 0x5577a725bc50 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.939+0000 7f82bde00640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a725bc50 0x5577a725e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.940+0000 7f82bde00640 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a725bc50 0x5577a725e040 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a668c2bfb77f35 server_cookie=3f78a01092c7bc52 in_seq=0 out_seq=0 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.940+0000 7f829fe00640 10 client.16050 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.940+0000 7f829fe00640 1 -- 172.21.15.63:0/219285370 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5577a71ba8b0 con 0x5577a725bc50 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f829fe00640 10 client.16050 handle_command_reply: tid=0 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 client.16050 shutdown 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 msgr2=0x5577a7248960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f82b8004e70 tx=0x7f82b8009be0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 shutdown_connections 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a725bc50 0x5577a725e040 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a72493e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 --2- 172.21.15.63:0/219285370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a7248960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f829fe00640 0 client.16050 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 >> 172.21.15.63:0/219285370 conn(0x5577a723cd30 msgr2=0x5577a7242a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 shutdown_connections 2024-04-25T18:55:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 1 -- 172.21.15.63:0/219285370 wait complete. 2024-04-25T18:55:11.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.941+0000 7f82c8218740 20 client.16050 trim_cache size 0 max 16384 2024-04-25T18:55:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.957+0000 7f82c8218740 1 Processor -- start 2024-04-25T18:55:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 -- start start 2024-04-25T18:55:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a7263760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b6c40 con 0x5577a7240820 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b44b0 con 0x5577a7241140 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82c8218740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b2320 con 0x5577a724ba00 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82bf200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33060/0 (socket says 172.21.15.63:33060) 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82bf200640 1 -- 172.21.15.63:0/4023268663 learned_addr learned my addr 172.21.15.63:0/4023268663 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82bf200640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a7263760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.958+0000 7f82bde00640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 -- 172.21.15.63:0/4023268663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 msgr2=0x5577a7263760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a7263760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 -- 172.21.15.63:0/4023268663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 msgr2=0x5577a7248240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 -- 172.21.15.63:0/4023268663 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577a724b1a0 con 0x5577a7240820 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bde00640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82bf200640 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f82b8011e50 tx=0x7f82b8009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993470109e83809c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82aaa00640 1 -- 172.21.15.63:0/4023268663 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b8028ac0 con 0x5577a7240820 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82aaa00640 1 -- 172.21.15.63:0/4023268663 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b802d070 con 0x5577a7240820 2024-04-25T18:55:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.959+0000 7f82aaa00640 1 -- 172.21.15.63:0/4023268663 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80046d0 con 0x5577a7240820 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- 172.21.15.63:0/4023268663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 msgr2=0x5577a72574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f82b8011e50 tx=0x7f82b8009230 comp rx=0 tx=0).stop 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- 172.21.15.63:0/4023268663 shutdown_connections 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7248240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a724ba00 0x5577a7263760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 --2- 172.21.15.63:0/4023268663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a7240820 0x5577a72574f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- 172.21.15.63:0/4023268663 >> 172.21.15.63:0/4023268663 conn(0x5577a723cd30 msgr2=0x5577a7242a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- 172.21.15.63:0/4023268663 shutdown_connections 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- 172.21.15.63:0/4023268663 wait complete. 2024-04-25T18:55:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.960+0000 7f82c8218740 1 -- start start 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 Processor -- start 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 0x5577a7250bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7251100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a724ba00 0x5577a72bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b2970 con 0x5577a724ba00 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b3de0 con 0x5577a7241140 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f82c01b3040 con 0x5577a7240820 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82c8218740 10 client.0 _ll_register_callbacks cb 0x7f82b8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82bf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 0x5577a7250bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82bf200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 0x5577a7250bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34852/0 (socket says 172.21.15.63:34852) 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82bf200640 1 -- 172.21.15.63:0/201988674 learned_addr learned my addr 172.21.15.63:0/201988674 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.961+0000 7f82bde00640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7251100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bf200640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a724ba00 0x5577a72bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 -- 172.21.15.63:0/201988674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 msgr2=0x5577a7250bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 0x5577a7250bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 -- 172.21.15.63:0/201988674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a724ba00 msgr2=0x5577a72bffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a724ba00 0x5577a72bffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 -- 172.21.15.63:0/201988674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577a72c0510 con 0x5577a7241140 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f82bde00640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7251100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009230 tx=0x7f82ac032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede166a19ab01553 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.962+0000 7f829fe00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac035040 con 0x5577a7241140 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82b802d070 con 0x5577a7241140 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 8631+0+0 (secure 0 0 0) 0x7f82ac004040 con 0x5577a7241140 2024-04-25T18:55:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f82ac054050 con 0x5577a7241140 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f82c8218740 10 client.26237 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5577a72c1830 con 0x5577a7241140 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.963+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b802d070 con 0x5577a7241140 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f82ac005040 con 0x5577a7241140 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 10 client.26237 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 20 client.26237 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a72c1fb0 0x5577a72c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 4 client.26237 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5577a71b4320 con 0x5577a72c1fb0 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82bf200640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a72c1fb0 0x5577a72c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f82bf200640 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a72c1fb0 0x5577a72c43d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d4e642f562984e server_cookie=359f537907c5c647 in_seq=0 out_seq=0 2024-04-25T18:55:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:11.964+0000 7f829fe00640 10 client.26237 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.011+0000 7f829fe00640 1 -- 172.21.15.63:0/201988674 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5577a71b4320 con 0x5577a72c1fb0 2024-04-25T18:55:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.011+0000 7f829fe00640 10 client.26237 handle_command_reply: tid=0 2024-04-25T18:55:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.011+0000 7f82c8218740 1 client.26237 shutdown 2024-04-25T18:55:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.011+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 msgr2=0x5577a7251100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.011+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7251100 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009230 tx=0x7f82ac032f40 comp rx=0 tx=0).stop 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 shutdown_connections 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5577a72c1fb0 0x5577a72c43d0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5577a7241140 0x5577a7251100 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5577a724ba00 0x5577a72bffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 --2- 172.21.15.63:0/201988674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5577a7240820 0x5577a7250bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f829fe00640 0 client.26237 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 >> 172.21.15.63:0/201988674 conn(0x5577a723cd30 msgr2=0x5577a7248920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 shutdown_connections 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 1 -- 172.21.15.63:0/201988674 wait complete. 2024-04-25T18:55:12.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.012+0000 7f82c8218740 20 client.26237 trim_cache size 0 max 16384 2024-04-25T18:55:12.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:55:12.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:55:12.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:55:12.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:55:12.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 msgr2=0x7f82a40a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 0x7f82a40a6280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82b00cdf90 tx=0x7f82b00d6550 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 msgr2=0x7f82c019ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c019ed80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f82b000e9f0 tx=0x7f82b000eeb0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 shutdown_connections 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f82a40a3dc0 0x7f82a40a6280 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f82c0112de0 0x7f82c01a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82c0076da0 0x7f82c019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 --2- 172.21.15.63:0/858230597 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f82c0076480 0x7f82c019ed80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 >> 172.21.15.63:0/858230597 conn(0x7f82c00ffc50 msgr2=0x7f82c0114570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 shutdown_connections 2024-04-25T18:55:12.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:12.013+0000 7f82c5400640 1 -- 172.21.15.63:0/858230597 wait complete. 2024-04-25T18:55:12.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.19 MB/sec execute 311 sec latency 8454.963 ms 2024-04-25T18:55:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:12 smithi114 ceph-mon[32988]: pgmap v268: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:12.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:12 smithi063 ceph-mon[28006]: pgmap v268: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:12 smithi070 ceph-mon[32820]: pgmap v268: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:13.484 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.18 MB/sec execute 312 sec latency 9455.101 ms 2024-04-25T18:55:14.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.16 MB/sec execute 313 sec latency 10455.248 ms 2024-04-25T18:55:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:14 smithi114 ceph-mon[32988]: pgmap v269: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:14 smithi114 ceph-mon[32988]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T18:55:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:14 smithi063 ceph-mon[28006]: pgmap v269: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:14.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:14 smithi063 ceph-mon[28006]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T18:55:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:14 smithi070 ceph-mon[32820]: pgmap v269: 97 pgs: 97 active+clean; 96 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.7 MiB/s wr, 160 op/s 2024-04-25T18:55:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:14 smithi070 ceph-mon[32820]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T18:55:15.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.15 MB/sec execute 314 sec latency 11455.411 ms 2024-04-25T18:55:16.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.14 MB/sec execute 315 sec latency 12455.591 ms 2024-04-25T18:55:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:16 smithi114 ceph-mon[32988]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:16 smithi063 ceph-mon[28006]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:16 smithi070 ceph-mon[32820]: pgmap v271: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:17.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.13 MB/sec execute 316 sec latency 13455.693 ms 2024-04-25T18:55:18.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.11 MB/sec execute 317 sec latency 14455.819 ms 2024-04-25T18:55:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:18 smithi114 ceph-mon[32988]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:18 smithi063 ceph-mon[28006]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:18.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:18 smithi070 ceph-mon[32820]: pgmap v272: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 1.0 MiB/s wr, 71 op/s 2024-04-25T18:55:19.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.10 MB/sec execute 318 sec latency 15455.910 ms 2024-04-25T18:55:20.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.09 MB/sec execute 319 sec latency 16456.017 ms 2024-04-25T18:55:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:20 smithi114 ceph-mon[32988]: pgmap v273: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:20 smithi063 ceph-mon[28006]: pgmap v273: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:20 smithi070 ceph-mon[32820]: pgmap v273: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:21.485 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.07 MB/sec execute 320 sec latency 17456.113 ms 2024-04-25T18:55:22.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.06 MB/sec execute 321 sec latency 18456.231 ms 2024-04-25T18:55:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:22 smithi114 ceph-mon[32988]: pgmap v274: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:22 smithi063 ceph-mon[28006]: pgmap v274: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:22 smithi070 ceph-mon[32820]: pgmap v274: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:23.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.05 MB/sec execute 322 sec latency 19456.854 ms 2024-04-25T18:55:24.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.04 MB/sec execute 323 sec latency 20457.021 ms 2024-04-25T18:55:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:24 smithi114 ceph-mon[32988]: pgmap v275: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:24 smithi063 ceph-mon[28006]: pgmap v275: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:24 smithi070 ceph-mon[32820]: pgmap v275: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 1.3 MiB/s wr, 62 op/s 2024-04-25T18:55:25.161 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 -- 172.21.15.63:0/1483123501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e410c330 msgr2=0x7f65e410e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 --2- 172.21.15.63:0/1483123501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e410c330 0x7f65e410e7d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f65dc009860 tx=0x7f65dc02a3d0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 -- 172.21.15.63:0/1483123501 shutdown_connections 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 --2- 172.21.15.63:0/1483123501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e410c330 0x7f65e410e7d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 --2- 172.21.15.63:0/1483123501 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 --2- 172.21.15.63:0/1483123501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e4102f10 0x7f65e41032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 -- 172.21.15.63:0/1483123501 >> 172.21.15.63:0/1483123501 conn(0x7f65e40fc770 msgr2=0x7f65e40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 -- 172.21.15.63:0/1483123501 shutdown_connections 2024-04-25T18:55:25.162 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.160+0000 7f65e3200640 1 -- 172.21.15.63:0/1483123501 wait complete. 2024-04-25T18:55:25.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.161+0000 7f65e3200640 1 Processor -- start 2024-04-25T18:55:25.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.161+0000 7f65e3200640 1 -- start start 2024-04-25T18:55:25.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.161+0000 7f65e3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e419ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.164 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e3200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e410c330 0x7f65e419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e3200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65e4111ff0 con 0x7f65e410c330 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e4111e70 con 0x7f65e4102f10 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65e4112170 con 0x7f65e4103830 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34084/0 (socket says 172.21.15.63:34084) 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 -- 172.21.15.63:0/2255081328 learned_addr learned my addr 172.21.15.63:0/2255081328 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e419ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e410c330 0x7f65e419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 -- 172.21.15.63:0/2255081328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 msgr2=0x7f65e419ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e419ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 -- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e410c330 msgr2=0x7f65e419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e410c330 0x7f65e419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.165 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1e00640 1 -- 172.21.15.63:0/2255081328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e419f740 con 0x7f65e4102f10 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.162+0000 7f65e1400640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65e1e00640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e9c0 tx=0x7f65cc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65daa00640 1 -- 172.21.15.63:0/2255081328 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00ccc0 con 0x7f65e4102f10 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65daa00640 1 -- 172.21.15.63:0/2255081328 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65cc0054d0 con 0x7f65e4102f10 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65e419fa30 con 0x7f65e4102f10 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65daa00640 1 -- 172.21.15.63:0/2255081328 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc010430 con 0x7f65e4102f10 2024-04-25T18:55:25.166 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.163+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65e41a72c0 con 0x7f65e4102f10 2024-04-25T18:55:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.166+0000 7f65ebfb0740 1 Processor -- start 2024-04-25T18:55:25.167 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 -- start start 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65daa00640 1 -- 172.21.15.63:0/2255081328 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f65cc004f80 con 0x7f65e4102f10 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a25d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a30650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56448298b900 con 0x564482a30b90 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564482931ef0 con 0x564482a262d0 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5644829878c0 con 0x564482a259b0 2024-04-25T18:55:25.169 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65e1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65e1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54924/0 (socket says 172.21.15.63:54924) 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65e1400640 1 -- 172.21.15.63:0/3683081067 learned_addr learned my addr 172.21.15.63:0/3683081067 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65e2800640 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a25d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.167+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a30650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 -- 172.21.15.63:0/3683081067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 msgr2=0x564482a25d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a25d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 -- 172.21.15.63:0/3683081067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 msgr2=0x564482a30650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a30650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 -- 172.21.15.63:0/3683081067 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56448269e210 con 0x564482a30b90 2024-04-25T18:55:25.170 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65e1400640 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f65d4011810 tx=0x7f65d4011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9322ec8167210467 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.173 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65d3400640 1 -- 172.21.15.63:0/3683081067 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d4004d20 con 0x564482a30b90 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65d3400640 1 -- 172.21.15.63:0/3683081067 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65d4003970 con 0x564482a30b90 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.168+0000 7f65d3400640 1 -- 172.21.15.63:0/3683081067 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d40044a0 con 0x564482a30b90 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65daa00640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 0x7f65c80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65daa00640 1 -- 172.21.15.63:0/2255081328 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f65cc0d0c80 con 0x7f65e4102f10 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65e2800640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 0x7f65c80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683081067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 msgr2=0x564482a32f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f65d4011810 tx=0x7f65d4011cd0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683081067 shutdown_connections 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a30650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a32f80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683081067 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a25d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.174 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683081067 >> 172.21.15.63:0/3683081067 conn(0x564482a21ec0 msgr2=0x564482a222a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.169+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683081067 shutdown_connections 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.170+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683081067 wait complete. 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.170+0000 7f65e2800640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 0x7f65c80a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f65dc032ff0 tx=0x7f65dc005c90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.170+0000 7f65ebfb0740 1 -- start start 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.170+0000 7f65ebfb0740 1 Processor -- start 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a2da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 0x564482a2dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564482a337c0 con 0x564482a262d0 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x564482945310 con 0x564482a30b90 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x564482a33940 con 0x564482a259b0 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65ebfb0740 10 client.0 _ll_register_callbacks cb 0x564482913cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:25.175 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 0x564482a2dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34096/0 (socket says 172.21.15.63:34096) 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e2800640 1 -- 172.21.15.63:0/3505450998 learned_addr learned my addr 172.21.15.63:0/3505450998 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a2da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 -- 172.21.15.63:0/3505450998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 msgr2=0x564482a2da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a2da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 -- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 msgr2=0x564482a2e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.171+0000 7f65e1400640 1 -- 172.21.15.63:0/3505450998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56448276a620 con 0x564482a262d0 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a2da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65e2800640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65e1400640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 0x564482a2dfa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f65d40117e0 tx=0x7f65d4004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68078ec2703e7ead server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.176 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d4027020 con 0x564482a262d0 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f65d400fa00 con 0x564482a262d0 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.172+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65d4003970 con 0x564482a262d0 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f65d4032050 con 0x564482a262d0 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 10 client.16068 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:25.177 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564482a3f990 con 0x564482a262d0 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65d4003970 con 0x564482a262d0 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f65d400fd20 con 0x564482a262d0 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 10 client.16068 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 10 client.16068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 20 client.16068 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482a40df0 0x564482a431e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 4 client.16068 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.173+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56448299f8b0 con 0x564482a40df0 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.174+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482a40df0 0x564482a431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.174+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482a40df0 0x564482a431e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=989179278b0ea3f5 server_cookie=14de2b3738563b9a in_seq=0 out_seq=0 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.174+0000 7f65c7e00640 10 client.16068 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65c7e00640 1 -- 172.21.15.63:0/3505450998 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56448299f8b0 con 0x564482a40df0 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65c7e00640 10 client.16068 handle_command_reply: tid=0 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 client.16068 shutdown 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 msgr2=0x564482a2dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 0x564482a2dfa0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f65d40117e0 tx=0x7f65d4004c70 comp rx=0 tx=0).stop 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 shutdown_connections 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482a40df0 0x564482a431e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a30b90 0x564482a2e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a262d0 0x564482a2dfa0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.180 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3505450998 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a2da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65c7e00640 0 client.16068 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 >> 172.21.15.63:0/3505450998 conn(0x564482a21ec0 msgr2=0x564482a27bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.175+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 shutdown_connections 2024-04-25T18:55:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.176+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3505450998 wait complete. 2024-04-25T18:55:25.181 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.176+0000 7f65ebfb0740 20 client.16068 trim_cache size 0 max 16384 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 Processor -- start 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 -- start start 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a48a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a48f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41b2670 con 0x564482a30b90 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41afee0 con 0x564482a262d0 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.192+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41add50 con 0x564482a259b0 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a48f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:32814/0 (socket says 172.21.15.63:32814) 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a48a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 -- 172.21.15.63:0/3683851241 learned_addr learned my addr 172.21.15.63:0/3683851241 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 -- 172.21.15.63:0/3683851241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 msgr2=0x564482a48a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a48a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 -- 172.21.15.63:0/3683851241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 msgr2=0x564482a48f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a48f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 -- 172.21.15.63:0/3683851241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564482a28870 con 0x564482a259b0 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.193+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ce390 tx=0x7f65cc0d4730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf3f45aba8f3f915 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65d2a00640 1 -- 172.21.15.63:0/3683851241 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc09c6c0 con 0x564482a259b0 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65d2a00640 1 -- 172.21.15.63:0/3683851241 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65cc09d280 con 0x564482a259b0 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65d2a00640 1 -- 172.21.15.63:0/3683851241 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00ce60 con 0x564482a259b0 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683851241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 msgr2=0x564482a3c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0ce390 tx=0x7f65cc0d4730 comp rx=0 tx=0).stop 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683851241 shutdown_connections 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a48a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.202 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a48f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3683851241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a3c680 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683851241 >> 172.21.15.63:0/3683851241 conn(0x564482a21ec0 msgr2=0x564482a28dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.194+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683851241 shutdown_connections 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3683851241 wait complete. 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 -- start start 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 Processor -- start 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a35b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a360a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41ae3a0 con 0x564482a30b90 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41af810 con 0x564482a262d0 2024-04-25T18:55:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.195+0000 7f65ebfb0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f65e41aea70 con 0x564482a259b0 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65ebfb0740 10 client.0 _ll_register_callbacks cb 0x7f65d4011660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54960/0 (socket says 172.21.15.63:54960) 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 -- 172.21.15.63:0/3452213186 learned_addr learned my addr 172.21.15.63:0/3452213186 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a35b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e1400640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a360a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 -- 172.21.15.63:0/3452213186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 msgr2=0x564482a35b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a35b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 -- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 msgr2=0x564482a360a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a360a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e2800640 1 -- 172.21.15.63:0/3452213186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564482aadbe0 con 0x564482a30b90 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a35b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.196+0000 7f65e1400640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a360a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65e2800640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f65dc02a9c0 tx=0x7f65dc02a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd35c81c36f6439b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65dc005510 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f65dc004020 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65cc09d280 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f65dc004220 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65ebfb0740 10 client.16074 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.197+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564482aaee70 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65cc09d280 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f65dc053070 con 0x564482a30b90 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 10 client.16074 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 10 client.16074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 20 client.16074 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482aaf670 0x564482ab1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 4 client.16074 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:55:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564482999320 con 0x564482aaf670 2024-04-25T18:55:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.198+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482aaf670 0x564482ab1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.199+0000 7f65e1e00640 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482aaf670 0x564482ab1a90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58eda2cccf015813 server_cookie=aa28d4ac94a47b18 in_seq=0 out_seq=0 2024-04-25T18:55:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.199+0000 7f65c7e00640 10 client.16074 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65c7e00640 1 -- 172.21.15.63:0/3452213186 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564482999320 con 0x564482aaf670 2024-04-25T18:55:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65c7e00640 10 client.16074 handle_command_reply: tid=0 2024-04-25T18:55:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 client.16074 shutdown 2024-04-25T18:55:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 msgr2=0x564482a365e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f65dc02a9c0 tx=0x7f65dc02a3d0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 shutdown_connections 2024-04-25T18:55:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x564482aaf670 0x564482ab1a90 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x564482a262d0 0x564482a360a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564482a30b90 0x564482a365e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65ebfb0740 1 --2- 172.21.15.63:0/3452213186 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x564482a259b0 0x564482a35b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.226+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 >> 172.21.15.63:0/3452213186 conn(0x564482a21ec0 msgr2=0x564482a2c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.225+0000 7f65c7e00640 0 client.16074 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.226+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 shutdown_connections 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.226+0000 7f65ebfb0740 1 -- 172.21.15.63:0/3452213186 wait complete. 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.226+0000 7f65ebfb0740 20 client.16074 trim_cache size 0 max 16384 2024-04-25T18:55:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:55:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:55:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:55:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:55:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 msgr2=0x7f65c80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 0x7f65c80a61d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f65dc032ff0 tx=0x7f65dc005c90 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 msgr2=0x7f65e419a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e9c0 tx=0x7f65cc00ee80 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 shutdown_connections 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f65c80a3d10 0x7f65c80a61d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65e410c330 0x7f65e419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f65e4103830 0x7f65e419ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 --2- 172.21.15.63:0/2255081328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65e4102f10 0x7f65e419a7a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 >> 172.21.15.63:0/2255081328 conn(0x7f65e40fc770 msgr2=0x7f65e4100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.228+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 shutdown_connections 2024-04-25T18:55:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:25.229+0000 7f65e3200640 1 -- 172.21.15.63:0/2255081328 wait complete. 2024-04-25T18:55:25.486 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.02 MB/sec execute 324 sec latency 21457.153 ms 2024-04-25T18:55:26.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.01 MB/sec execute 325 sec latency 22457.227 ms 2024-04-25T18:55:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:26 smithi114 ceph-mon[32988]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-04-25T18:55:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:26 smithi063 ceph-mon[28006]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-04-25T18:55:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:26 smithi070 ceph-mon[32820]: pgmap v276: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-04-25T18:55:27.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 4.00 MB/sec execute 326 sec latency 23457.352 ms 2024-04-25T18:55:28.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.99 MB/sec execute 327 sec latency 24457.494 ms 2024-04-25T18:55:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:28 smithi114 ceph-mon[32988]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-25T18:55:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:28 smithi063 ceph-mon[28006]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-25T18:55:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:28 smithi070 ceph-mon[32820]: pgmap v277: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 1.1 MiB/s wr, 44 op/s 2024-04-25T18:55:29.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.97 MB/sec execute 328 sec latency 25457.642 ms 2024-04-25T18:55:29.687 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:55:30.152 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:55:30.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.96 MB/sec execute 329 sec latency 26457.745 ms 2024-04-25T18:55:30.534 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:30 smithi063 ceph-mon[28006]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 69 op/s 2024-04-25T18:55:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:30 smithi114 ceph-mon[32988]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 69 op/s 2024-04-25T18:55:30.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:30 smithi070 ceph-mon[32820]: pgmap v278: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 1.4 MiB/s wr, 69 op/s 2024-04-25T18:55:31.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.95 MB/sec execute 330 sec latency 27457.856 ms 2024-04-25T18:55:32.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 -- 172.21.15.63:0/2289480841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 msgr2=0x7fa5c0111860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2289480841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c0111860 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa5b8009f90 tx=0x7fa5b800b330 comp rx=0 tx=0).stop 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 -- 172.21.15.63:0/2289480841 shutdown_connections 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2289480841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2289480841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c0111860 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2289480841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c010ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 -- 172.21.15.63:0/2289480841 >> 172.21.15.63:0/2289480841 conn(0x7fa5c0079b10 msgr2=0x7fa5c0079f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.444+0000 7fa5c7800700 1 -- 172.21.15.63:0/2289480841 shutdown_connections 2024-04-25T18:55:32.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.445+0000 7fa5c7800700 1 -- 172.21.15.63:0/2289480841 wait complete. 2024-04-25T18:55:32.448 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:32 smithi063 ceph-mon[28006]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.446+0000 7fa5c7800700 1 Processor -- start 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.446+0000 7fa5c7800700 1 -- start start 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c019b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0114e50 con 0x7fa5c0075590 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0114cd0 con 0x7fa5c010f470 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c7800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c0114fd0 con 0x7fa5c0074c70 2024-04-25T18:55:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5bfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c4e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c019b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5bfe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34908/0 (socket says 172.21.15.63:34908) 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5bfe00700 1 -- 172.21.15.63:0/2124996716 learned_addr learned my addr 172.21.15.63:0/2124996716 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.447+0000 7fa5c5800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 -- 172.21.15.63:0/2124996716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 msgr2=0x7fa5c019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 -- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 msgr2=0x7fa5c019b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 -- 172.21.15.63:0/2124996716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c01a0300 con 0x7fa5c0074c70 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5bfe00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c4e00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c019b340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e9f0 tx=0x7fa5b400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400c9c0 con 0x7fa5c0074c70 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c01a05f0 con 0x7fa5c0074c70 2024-04-25T18:55:32.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5b400cb60 con 0x7fa5c0074c70 2024-04-25T18:55:32.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.448+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4018680 con 0x7fa5c0074c70 2024-04-25T18:55:32.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.449+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c01a7e70 con 0x7fa5c0074c70 2024-04-25T18:55:32.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.450+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c0069730 con 0x7fa5c0074c70 2024-04-25T18:55:32.452 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.452+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa5b4010430 con 0x7fa5c0074c70 2024-04-25T18:55:32.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.453+0000 7fa5bd600700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 0x7fa5a80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:32.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.453+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fa5b40d0ac0 con 0x7fa5c0074c70 2024-04-25T18:55:32.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.453+0000 7fa5bfe00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 0x7fa5a80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:32.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.454+0000 7fa5bfe00700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 0x7fa5a80a62f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009750 tx=0x7fa5ac009310 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:32.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.455+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5b409d0c0 con 0x7fa5c0074c70 2024-04-25T18:55:32.487 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.94 MB/sec execute 331 sec latency 28458.000 ms 2024-04-25T18:55:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:32 smithi114 ceph-mon[32988]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:32 smithi070 ceph-mon[32820]: pgmap v279: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:32.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.736+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5c019c650 con 0x7fa5c0074c70 2024-04-25T18:55:32.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.737+0000 7fa5bd600700 1 -- 172.21.15.63:0/2124996716 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11871 (secure 0 0 0) 0x7fa5b409c9c0 con 0x7fa5c0074c70 2024-04-25T18:55:32.739 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:55:32.739 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:53:53.997852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T18:55:32.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 msgr2=0x7fa5a80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:32.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 0x7fa5a80a62f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac009750 tx=0x7fa5ac009310 comp rx=0 tx=0).stop 2024-04-25T18:55:32.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 msgr2=0x7fa5c019b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c019b340 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e9f0 tx=0x7fa5b400ed00 comp rx=0 tx=0).stop 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 shutdown_connections 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5a80a3e30 0x7fa5a80a62f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa5c0074c70 0x7fa5c019b340 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5c010f470 0x7fa5c019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 --2- 172.21.15.63:0/2124996716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5c0075590 0x7fa5c019b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 >> 172.21.15.63:0/2124996716 conn(0x7fa5c0079b10 msgr2=0x7fa5c0100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 shutdown_connections 2024-04-25T18:55:32.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:32.742+0000 7fa5c7800700 1 -- 172.21.15.63:0/2124996716 wait complete. 2024-04-25T18:55:32.745 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 34 2024-04-25T18:55:33.488 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.93 MB/sec execute 332 sec latency 29458.102 ms 2024-04-25T18:55:33.564 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:33 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2124996716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:33.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:33 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2124996716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:33.632 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id 2a159cfd --release --await' 2024-04-25T18:55:33.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id 2a159cfd --release --await 2024-04-25T18:55:33.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:33 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2124996716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:34.115 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:55:34.488 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.91 MB/sec execute 333 sec latency 30458.798 ms 2024-04-25T18:55:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:34 smithi114 ceph-mon[32988]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:34.567 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:34 smithi063 ceph-mon[28006]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:34 smithi070 ceph-mon[32820]: pgmap v280: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 839 KiB/s wr, 51 op/s 2024-04-25T18:55:35.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306574 3.90 MB/sec execute 334 sec latency 31463.393 ms 2024-04-25T18:55:36.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 -- 172.21.15.63:0/1900909676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df4074c70 msgr2=0x7f7df4075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 --2- 172.21.15.63:0/1900909676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df4074c70 0x7f7df4075050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7de400b0a0 tx=0x7f7de400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 -- 172.21.15.63:0/1900909676 shutdown_connections 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 --2- 172.21.15.63:0/1900909676 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df410f4a0 0x7f7df4111890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 --2- 172.21.15.63:0/1900909676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df4074c70 0x7f7df4075050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 --2- 172.21.15.63:0/1900909676 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 0x7f7df410ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 -- 172.21.15.63:0/1900909676 >> 172.21.15.63:0/1900909676 conn(0x7f7df4079b90 msgr2=0x7f7df4079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 -- 172.21.15.63:0/1900909676 shutdown_connections 2024-04-25T18:55:36.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.339+0000 7f7dfa000700 1 -- 172.21.15.63:0/1900909676 wait complete. 2024-04-25T18:55:36.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.340+0000 7f7dfa000700 1 Processor -- start 2024-04-25T18:55:36.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.340+0000 7f7dfa000700 1 -- start start 2024-04-25T18:55:36.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 0x7f7df419b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df410f4a0 0x7f7df419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4114f20 con 0x7f7df410f4a0 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4114da0 con 0x7f7df4074c70 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7dfa000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41150a0 con 0x7f7df4075590 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7df3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df410f4a0 0x7f7df419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 0x7f7df419b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41790/0 (socket says 172.21.15.63:41790) 2024-04-25T18:55:36.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7df3400700 1 -- 172.21.15.63:0/3621571359 learned_addr learned my addr 172.21.15.63:0/3621571359 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:36.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.341+0000 7f7df3400700 1 -- 172.21.15.63:0/3621571359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 msgr2=0x7f7df419b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df3400700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 0x7f7df419b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df3400700 1 -- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df410f4a0 msgr2=0x7f7df419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df3400700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df410f4a0 0x7f7df419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df3400700 1 -- 172.21.15.63:0/3621571359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df41a0420 con 0x7f7df4074c70 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df3400700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7de400d010 tx=0x7f7de4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df0c00700 1 -- 172.21.15.63:0/3621571359 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4021070 con 0x7f7df4074c70 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df0c00700 1 -- 172.21.15.63:0/3621571359 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7de4007e30 con 0x7f7df4074c70 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df41a06b0 con 0x7f7df4074c70 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7df0c00700 1 -- 172.21.15.63:0/3621571359 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4004640 con 0x7f7df4074c70 2024-04-25T18:55:36.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.342+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df41a7f90 con 0x7f7df4074c70 2024-04-25T18:55:36.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.346+0000 7f7df0c00700 1 -- 172.21.15.63:0/3621571359 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7de4007980 con 0x7f7df4074c70 2024-04-25T18:55:36.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.347+0000 7f7df0c00700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 0x7f7dd80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.347+0000 7f7df0c00700 1 -- 172.21.15.63:0/3621571359 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f7de400f020 con 0x7f7df4074c70 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.347+0000 7f7deaa00700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 0x7f7dd80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 Processor -- start 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 -- start start 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7deaa00700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 0x7f7dd80a62f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7df4102640 tx=0x7f7ddc009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb18120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb258c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56450fa6af90 con 0x56450fb234d0 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56450fa11830 con 0x56450fb17d40 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.348+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56450fa66f50 con 0x56450fb18660 2024-04-25T18:55:36.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb18120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44798/0 (socket says 172.21.15.63:44798) 2024-04-25T18:55:36.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 -- 172.21.15.63:0/2588694906 learned_addr learned my addr 172.21.15.63:0/2588694906 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:36.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7df3e00700 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb258c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 -- 172.21.15.63:0/2588694906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 msgr2=0x56450fb18120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb18120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 -- 172.21.15.63:0/2588694906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 msgr2=0x56450fb258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb258c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 -- 172.21.15.63:0/2588694906 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56450fa07780 con 0x56450fb18660 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7df3400700 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb18120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.349+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc011da0 tx=0x7f7ddc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bf85480890a970 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.350+0000 7f7de8c00700 1 -- 172.21.15.63:0/2588694906 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc004a30 con 0x56450fb18660 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.350+0000 7f7de8c00700 1 -- 172.21.15.63:0/2588694906 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc0054d0 con 0x56450fb18660 2024-04-25T18:55:36.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.350+0000 7f7de8c00700 1 -- 172.21.15.63:0/2588694906 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc003b20 con 0x56450fb18660 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.350+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2588694906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 msgr2=0x56450fb22f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.350+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc011da0 tx=0x7f7ddc014000 comp rx=0 tx=0).stop 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2588694906 shutdown_connections 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb258c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb18120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2588694906 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb22f90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2588694906 >> 172.21.15.63:0/2588694906 conn(0x56450fa47050 msgr2=0x56450fb14250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2588694906 shutdown_connections 2024-04-25T18:55:36.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2588694906 wait complete. 2024-04-25T18:55:36.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.351+0000 7f7e0490ab80 1 -- start start 2024-04-25T18:55:36.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 Processor -- start 2024-04-25T18:55:36.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 0x56450fb22bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb234d0 0x56450fb20410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56450fb25fc0 con 0x56450fb17d40 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56450fa24960 con 0x56450fb234d0 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56450fb26140 con 0x56450fb18660 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7e0490ab80 10 client.0 _ll_register_callbacks cb 0x56450fa03820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:36.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7df3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 0x56450fb22bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.352+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44814/0 (socket says 172.21.15.63:44814) 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 -- 172.21.15.63:0/2006386992 learned_addr learned my addr 172.21.15.63:0/2006386992 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7df3e00700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb234d0 0x56450fb20410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 -- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb234d0 msgr2=0x56450fb20410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb234d0 0x56450fb20410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 -- 172.21.15.63:0/2006386992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 msgr2=0x56450fb22bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 0x56450fb22bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 -- 172.21.15.63:0/2006386992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56450f9b89d0 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7df3400700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 0x56450fb22bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009840 tx=0x7f7ddc0048e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbd3ab463c02477 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.353+0000 7f7de1600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc005680 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f7ddc003720 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc0054d0 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc003920 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7e0490ab80 10 client.26299 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56450fa01bf0 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.354+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc0054d0 con 0x56450fb18660 2024-04-25T18:55:36.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f7ddc037030 con 0x56450fb18660 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 10 client.26299 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 10 client.26299 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 20 client.26299 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb32be0 0x56450fb34fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 4 client.26299 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56450fa78780 con 0x56450fb32be0 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.355+0000 7f7df3400700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb32be0 0x56450fb34fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.356+0000 7f7df3400700 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb32be0 0x56450fb34fd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ab6622e2d2ec37 server_cookie=a6412b43e7fd0562 in_seq=0 out_seq=0 2024-04-25T18:55:36.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.356+0000 7f7de1600700 10 client.26299 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7de1600700 1 -- 172.21.15.63:0/2006386992 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56450fa78780 con 0x56450fb32be0 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7de1600700 10 client.26299 handle_command_reply: tid=0 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 client.26299 shutdown 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 msgr2=0x56450fb1fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009840 tx=0x7f7ddc0048e0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 shutdown_connections 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb32be0 0x56450fb34fd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb234d0 0x56450fb20410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb17d40 0x56450fb22bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2006386992 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb1fed0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 >> 172.21.15.63:0/2006386992 conn(0x56450fa47050 msgr2=0x56450fb13e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7de1600700 0 client.26299 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 shutdown_connections 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.357+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2006386992 wait complete. 2024-04-25T18:55:36.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.358+0000 7f7e0490ab80 20 client.26299 trim_cache size 0 max 16384 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 Processor -- start 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 -- start start 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb3a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb3aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb2b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41b1ef0 con 0x56450fb234d0 2024-04-25T18:55:36.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41af720 con 0x56450fb17d40 2024-04-25T18:55:36.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.379+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41ad590 con 0x56450fb18660 2024-04-25T18:55:36.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7df3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb2b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb3aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7df3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb2b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34948/0 (socket says 172.21.15.63:34948) 2024-04-25T18:55:36.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb3a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7df3e00700 1 -- 172.21.15.63:0/2585578317 learned_addr learned my addr 172.21.15.63:0/2585578317 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:36.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 -- 172.21.15.63:0/2585578317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 msgr2=0x56450fb3a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb3a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 -- 172.21.15.63:0/2585578317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 msgr2=0x56450fb2b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb2b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.380+0000 7f7deaa00700 1 -- 172.21.15.63:0/2585578317 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56450fb18ce0 con 0x56450fb18660 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.381+0000 7f7deaa00700 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb3aab0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0117d0 tx=0x7f7ddc004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7d7d743178f112 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.381+0000 7f7de3e00700 1 -- 172.21.15.63:0/2585578317 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc031b20 con 0x56450fb18660 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.381+0000 7f7de3e00700 1 -- 172.21.15.63:0/2585578317 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc031cc0 con 0x56450fb18660 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.381+0000 7f7de3e00700 1 -- 172.21.15.63:0/2585578317 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc031680 con 0x56450fb18660 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.381+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2585578317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 msgr2=0x56450fb3aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb3aab0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0117d0 tx=0x7f7ddc004970 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2585578317 shutdown_connections 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb2b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb3a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/2585578317 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb3aab0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2585578317 >> 172.21.15.63:0/2585578317 conn(0x56450fa47050 msgr2=0x56450fb1fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2585578317 shutdown_connections 2024-04-25T18:55:36.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 -- 172.21.15.63:0/2585578317 wait complete. 2024-04-25T18:55:36.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 -- start start 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.382+0000 7f7e0490ab80 1 Processor -- start 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb2b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb27f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41adbe0 con 0x56450fb234d0 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41af050 con 0x56450fb17d40 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41ae2b0 con 0x56450fb18660 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7e0490ab80 10 client.0 _ll_register_callbacks cb 0x56450fa03820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:36.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb2b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7df3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb27f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7df3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb2b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41834/0 (socket says 172.21.15.63:41834) 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7deaa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44828/0 (socket says 172.21.15.63:44828) 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.383+0000 7f7df3400700 1 -- 172.21.15.63:0/4172983625 learned_addr learned my addr 172.21.15.63:0/4172983625 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 -- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 msgr2=0x56450fb2b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb2b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 -- 172.21.15.63:0/4172983625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 msgr2=0x56450fb27f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb27f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 -- 172.21.15.63:0/4172983625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56450fb28650 con 0x56450fb18660 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7deaa00700 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x56450fb18af0 tx=0x7f7ddc005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68fe7fce14f90f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7de1600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.384+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0053f0 con 0x56450fb18660 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc031cc0 con 0x56450fb18660 2024-04-25T18:55:36.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f7ddc005590 con 0x56450fb18660 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc003ac0 con 0x56450fb18660 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 10 client.26305 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56450fb99d20 con 0x56450fb18660 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc031cc0 con 0x56450fb18660 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7f7ddc048ba0 con 0x56450fb18660 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 10 client.26305 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 10 client.26305 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 20 client.26305 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb9a260 0x56450fb9c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 4 client.26305 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2a159cfd", "release": true, "await": true}] 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.385+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2a159cfd", "release": true, "await": true}) -- 0x56450fa72820 con 0x56450fb9a260 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.386+0000 7f7df3400700 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb9a260 0x56450fb9c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:36.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.386+0000 7f7df3400700 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb9a260 0x56450fb9c650 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b467ba8f3a89e13 server_cookie=2469822b041c4a14 in_seq=0 out_seq=0 2024-04-25T18:55:36.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.386+0000 7f7de1600700 10 client.26305 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:36.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7de1600700 1 -- 172.21.15.63:0/4172983625 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56450fa72820 con 0x56450fb9a260 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7de1600700 10 client.26305 handle_command_reply: tid=0 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 client.26305 shutdown 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 msgr2=0x56450fb27a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x56450fb18af0 tx=0x7f7ddc005a00 comp rx=0 tx=0).stop 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 shutdown_connections 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56450fb9a260 0x56450fb9c650 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56450fb234d0 0x56450fb27f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56450fb17d40 0x56450fb2b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 --2- 172.21.15.63:0/4172983625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56450fb18660 0x56450fb27a10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 >> 172.21.15.63:0/4172983625 conn(0x56450fa47050 msgr2=0x56450fb21700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7de1600700 0 client.26305 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:36.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 shutdown_connections 2024-04-25T18:55:36.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 1 -- 172.21.15.63:0/4172983625 wait complete. 2024-04-25T18:55:36.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.391+0000 7f7e0490ab80 20 client.26305 trim_cache size 0 max 16384 2024-04-25T18:55:36.394 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:55:36.394 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 34, 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 28, 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "2a159cfd": { 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 28, 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:36.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 84.8, 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:36.396 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 msgr2=0x7f7dd80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 0x7f7dd80a62f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7df4102640 tx=0x7f7ddc009260 comp rx=0 tx=0).stop 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 msgr2=0x7f7df419b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:36.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7de400d010 tx=0x7f7de4004d00 comp rx=0 tx=0).stop 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 shutdown_connections 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7dd80a3e30 0x7f7dd80a62f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7df410f4a0 0x7f7df419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7df4074c70 0x7f7df419b410 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 --2- 172.21.15.63:0/3621571359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7df4075590 0x7f7df419b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.392+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 >> 172.21.15.63:0/3621571359 conn(0x7f7df4079b90 msgr2=0x7f7df40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.393+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 shutdown_connections 2024-04-25T18:55:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:36.393+0000 7f7dfa000700 1 -- 172.21.15.63:0/3621571359 wait complete. 2024-04-25T18:55:36.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 306648 3.89 MB/sec execute 335 sec latency 32369.615 ms 2024-04-25T18:55:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:36 smithi114 ceph-mon[32988]: pgmap v281: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:55:36.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:36 smithi063 ceph-mon[28006]: pgmap v281: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:55:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:36 smithi070 ceph-mon[32820]: pgmap v281: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-04-25T18:55:37.347 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2a159cfd', seconds elapsed: 41.1 2024-04-25T18:55:37.347 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 24.8 2024-04-25T18:55:37.347 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:55:37.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 308039 3.90 MB/sec execute 336 sec latency 21.305 ms 2024-04-25T18:55:37.846 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 -- 172.21.15.63:0/3779190993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 msgr2=0x7faeb00746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 --2- 172.21.15.63:0/3779190993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb00746d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7faea4009900 tx=0x7faea402a480 comp rx=0 tx=0).stop 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 -- 172.21.15.63:0/3779190993 shutdown_connections 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 --2- 172.21.15.63:0/3779190993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0112f40 0x7faeb0115330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 --2- 172.21.15.63:0/3779190993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0074c10 0x7faeb0112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 --2- 172.21.15.63:0/3779190993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb00746d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.406+0000 7faeb8600640 1 -- 172.21.15.63:0/3779190993 >> 172.21.15.63:0/3779190993 conn(0x7faeb00fda00 msgr2=0x7faeb00ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.407+0000 7faeb8600640 1 -- 172.21.15.63:0/3779190993 shutdown_connections 2024-04-25T18:55:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.407+0000 7faeb8600640 1 -- 172.21.15.63:0/3779190993 wait complete. 2024-04-25T18:55:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.408+0000 7faeb8600640 1 Processor -- start 2024-04-25T18:55:38.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.408+0000 7faeb8600640 1 -- start start 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0112f40 0x7faeb019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0118810 con 0x7faeb0112f40 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0118690 con 0x7faeb00742f0 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb8600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0118990 con 0x7faeb0074c10 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49122/0 (socket says 172.21.15.63:49122) 2024-04-25T18:55:38.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57996/0 (socket says 172.21.15.63:57996) 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb5c00640 1 -- 172.21.15.63:0/3940780029 learned_addr learned my addr 172.21.15.63:0/3940780029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.409+0000 7faeb6600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0112f40 0x7faeb019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 -- 172.21.15.63:0/3940780029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 msgr2=0x7faeb01a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 -- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0112f40 msgr2=0x7faeb019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0112f40 0x7faeb019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 -- 172.21.15.63:0/3940780029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb019f270 con 0x7faeb00742f0 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5200640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:38.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb5c00640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faea4005ab0 tx=0x7faea4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeae000640 1 -- 172.21.15.63:0/3940780029 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea403f070 con 0x7faeb00742f0 2024-04-25T18:55:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeae000640 1 -- 172.21.15.63:0/3940780029 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea4031b60 con 0x7faeb00742f0 2024-04-25T18:55:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeae000640 1 -- 172.21.15.63:0/3940780029 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea4035030 con 0x7faeb00742f0 2024-04-25T18:55:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.410+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb019f500 con 0x7faeb00742f0 2024-04-25T18:55:38.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.411+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb01ab800 con 0x7faeb00742f0 2024-04-25T18:55:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.415+0000 7faebb36b740 1 Processor -- start 2024-04-25T18:55:38.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.415+0000 7faebb36b740 1 -- start start 2024-04-25T18:55:38.417 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeae000640 1 -- 172.21.15.63:0/3940780029 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7faea4042070 con 0x7faeb00742f0 2024-04-25T18:55:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde403e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.418 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde4040df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cde3f99900 con 0x55cde403ea00 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cde3f3fef0 con 0x55cde4033820 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faebb36b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cde3f958c0 con 0x55cde4034140 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeb5c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde403e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeb6600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeb6600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49136/0 (socket says 172.21.15.63:49136) 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeb6600640 1 -- 172.21.15.63:0/2891187721 learned_addr learned my addr 172.21.15.63:0/2891187721 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.416+0000 7faeb5200640 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde4040df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 -- 172.21.15.63:0/2891187721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 msgr2=0x55cde403e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde403e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 -- 172.21.15.63:0/2891187721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 msgr2=0x55cde4040df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde4040df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 -- 172.21.15.63:0/2891187721 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde3cac210 con 0x55cde4033820 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faeb6600640 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faea8009f00 tx=0x7faea80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc9ea5b8f4ad0544 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.419 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faea2a00640 1 -- 172.21.15.63:0/2891187721 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8003b70 con 0x55cde4033820 2024-04-25T18:55:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faea2a00640 1 -- 172.21.15.63:0/2891187721 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea8003d10 con 0x55cde4033820 2024-04-25T18:55:38.420 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.417+0000 7faea2a00640 1 -- 172.21.15.63:0/2891187721 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8011980 con 0x55cde4033820 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faeae000640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 0x7fae980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 -- 172.21.15.63:0/2891187721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 msgr2=0x55cde4033c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faea8009f00 tx=0x7faea80107a0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faeae000640 1 -- 172.21.15.63:0/3940780029 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7faea40f0e90 con 0x7faeb00742f0 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 -- 172.21.15.63:0/2891187721 shutdown_connections 2024-04-25T18:55:38.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde4040df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde403e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 --2- 172.21.15.63:0/2891187721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4033820 0x55cde4033c00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faebb36b740 1 -- 172.21.15.63:0/2891187721 >> 172.21.15.63:0/2891187721 conn(0x55cde402fd30 msgr2=0x55cde4030110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.418+0000 7faeb5c00640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 0x7fae980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.419+0000 7faebb36b740 1 -- 172.21.15.63:0/2891187721 shutdown_connections 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.419+0000 7faebb36b740 1 -- 172.21.15.63:0/2891187721 wait complete. 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.419+0000 7faebb36b740 1 -- start start 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.419+0000 7faeb5c00640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 0x7fae980a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faea40ee650 tx=0x7faea40f49e0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 Processor -- start 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4033820 0x55cde403b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde403c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cde4041630 con 0x55cde403ea00 2024-04-25T18:55:38.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55cde3f53310 con 0x55cde4034140 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55cde40417b0 con 0x55cde4033820 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faebb36b740 10 client.0 _ll_register_callbacks cb 0x55cde3f21cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.420+0000 7faeb5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49150/0 (socket says 172.21.15.63:49150) 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 -- 172.21.15.63:0/1914741514 learned_addr learned my addr 172.21.15.63:0/1914741514 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb6600640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4033820 0x55cde403b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb6600640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde403c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 -- 172.21.15.63:0/1914741514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4033820 msgr2=0x55cde403b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4033820 0x55cde403b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 -- 172.21.15.63:0/1914741514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 msgr2=0x55cde403c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde403c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 -- 172.21.15.63:0/1914741514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde3d78620 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7faeb5200640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fae9c012a10 tx=0x7fae9c012ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31749d67f8426e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.421+0000 7fae97400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c004050 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faea8003d10 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fae9c0041f0 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fae9c0043f0 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7faebb36b740 10 client.26291 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cde404d9a0 con 0x55cde4034140 2024-04-25T18:55:38.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.422+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea8003d10 con 0x55cde4034140 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fae9c003c00 con 0x55cde4034140 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 10 client.26291 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 10 client.26291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 20 client.26291 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde4051250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 4 client.26291 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cde3fad8b0 con 0x55cde404ee60 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faeb6600640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde4051250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7fae97400640 10 client.26291 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.423+0000 7faeb6600640 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde4051250 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b2b00b65bce02f6 server_cookie=29dc48efd30051f7 in_seq=0 out_seq=0 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7fae97400640 1 -- 172.21.15.63:0/1914741514 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cde3fad8b0 con 0x55cde404ee60 2024-04-25T18:55:38.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7fae97400640 10 client.26291 handle_command_reply: tid=0 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 client.26291 shutdown 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 msgr2=0x55cde403bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fae9c012a10 tx=0x7fae9c012ed0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 shutdown_connections 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde4051250 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7fae97400640 0 client.26291 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde403bef0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde403ea00 0x55cde403c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 --2- 172.21.15.63:0/1914741514 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4033820 0x55cde403b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 >> 172.21.15.63:0/1914741514 conn(0x55cde402fd30 msgr2=0x55cde4035a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 shutdown_connections 2024-04-25T18:55:38.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.425+0000 7faebb36b740 1 -- 172.21.15.63:0/1914741514 wait complete. 2024-04-25T18:55:38.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.426+0000 7faebb36b740 20 client.26291 trim_cache size 0 max 16384 2024-04-25T18:55:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.451+0000 7faebb36b740 1 Processor -- start 2024-04-25T18:55:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.451+0000 7faebb36b740 1 -- start start 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde404a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 0x55cde4056b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b6b40 con 0x55cde4033820 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b43b0 con 0x55cde403ea00 2024-04-25T18:55:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.452+0000 7faebb36b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b2220 con 0x55cde4034140 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb6600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde404a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58010/0 (socket says 172.21.15.63:58010) 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 -- 172.21.15.63:0/2731805129 learned_addr learned my addr 172.21.15.63:0/2731805129 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb6600640 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 0x55cde4056b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 -- 172.21.15.63:0/2731805129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 msgr2=0x55cde4056b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 0x55cde4056b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 -- 172.21.15.63:0/2731805129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 msgr2=0x55cde404a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde404a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 -- 172.21.15.63:0/2731805129 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cde40515c0 con 0x55cde4034140 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb6600640 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 0x55cde4056b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.453+0000 7faeb5200640 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001eb0 tx=0x7fae9c0123e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68708c734869259d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.454+0000 7faea2000640 1 -- 172.21.15.63:0/2731805129 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c031070 con 0x55cde4034140 2024-04-25T18:55:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.454+0000 7faea2000640 1 -- 172.21.15.63:0/2731805129 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9c0106c0 con 0x55cde4034140 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.454+0000 7faea2000640 1 -- 172.21.15.63:0/2731805129 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c02c690 con 0x55cde4034140 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 -- 172.21.15.63:0/2731805129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 msgr2=0x55cde402ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fae9c001eb0 tx=0x7fae9c0123e0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 -- 172.21.15.63:0/2731805129 shutdown_connections 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde403ea00 0x55cde4056b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde4034140 0x55cde402ed10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 --2- 172.21.15.63:0/2731805129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde404a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 -- 172.21.15.63:0/2731805129 >> 172.21.15.63:0/2731805129 conn(0x55cde402fd30 msgr2=0x55cde402e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 -- 172.21.15.63:0/2731805129 shutdown_connections 2024-04-25T18:55:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.455+0000 7faebb36b740 1 -- 172.21.15.63:0/2731805129 wait complete. 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.456+0000 7faebb36b740 1 -- start start 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.456+0000 7faebb36b740 1 Processor -- start 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.456+0000 7faebb36b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.456+0000 7faebb36b740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde4044960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faebb36b740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde403ea00 0x55cde4044ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faebb36b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b2870 con 0x55cde4033820 2024-04-25T18:55:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faebb36b740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b3ce0 con 0x55cde4034140 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faebb36b740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01b2f40 con 0x55cde403ea00 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faebb36b740 10 client.0 _ll_register_callbacks cb 0x7fae9c012620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faeb6600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faeb6600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35620/0 (socket says 172.21.15.63:35620) 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faeb6600640 1 -- 172.21.15.63:0/3542635102 learned_addr learned my addr 172.21.15.63:0/3542635102 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faeb5200640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde4044960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.457+0000 7faeb6600640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde403ea00 0x55cde4044ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 -- 172.21.15.63:0/3542635102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde403ea00 msgr2=0x55cde4044ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde403ea00 0x55cde4044ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 -- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 msgr2=0x55cde4044960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde4044960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 -- 172.21.15.63:0/3542635102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cde40b43a0 con 0x55cde4033820 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7faeb6600640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faea8002230 tx=0x7faea801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175908194ee1bd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.458+0000 7fae97400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8025040 con 0x55cde4033820 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7faea8013c40 con 0x55cde4033820 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae9c0106c0 con 0x55cde4033820 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7faea8013e40 con 0x55cde4033820 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7faebb36b740 10 client.16116 fetch_fsmap learned FSMap version 34 2024-04-25T18:55:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.459+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cde40b5680 con 0x55cde4033820 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9c0106c0 con 0x55cde4033820 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 34) ==== 8346+0+0 (secure 0 0 0) 0x7fae9c009ce0 con 0x55cde4033820 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7faebb36b740 10 client.16116 fetch_fsmap finished waiting for FSMap version 34 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7faebb36b740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7faebb36b740 20 client.16116 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.460+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde40b7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.461+0000 7faebb36b740 4 client.16116 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.461+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cde3fa7320 con 0x55cde404ee60 2024-04-25T18:55:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.461+0000 7faeb5200640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde40b7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.462+0000 7faeb5200640 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde40b7bc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b29014bf585fa22 server_cookie=70483204e678bfdd in_seq=0 out_seq=0 2024-04-25T18:55:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.462+0000 7fae97400640 10 client.16116 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:38.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 309071 3.91 MB/sec execute 337 sec latency 18.342 ms 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7fae97400640 1 -- 172.21.15.63:0/3542635102 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cde3fa7320 con 0x55cde404ee60 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7fae97400640 10 client.16116 handle_command_reply: tid=0 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 client.16116 shutdown 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 msgr2=0x55cde4044420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7faea8002230 tx=0x7faea801e040 comp rx=0 tx=0).stop 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 shutdown_connections 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55cde404ee60 0x55cde40b7bc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55cde4034140 0x55cde4044960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7fae97400640 0 client.16116 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55cde403ea00 0x55cde4044ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 --2- 172.21.15.63:0/3542635102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cde4033820 0x55cde4044420 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 >> 172.21.15.63:0/3542635102 conn(0x55cde402fd30 msgr2=0x55cde402ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 shutdown_connections 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.508+0000 7faebb36b740 1 -- 172.21.15.63:0/3542635102 wait complete. 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faebb36b740 20 client.16116 trim_cache size 0 max 16384 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:55:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 msgr2=0x7fae980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 0x7fae980a6300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7faea40ee650 tx=0x7faea40f49e0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 msgr2=0x7faeb01a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faea4005ab0 tx=0x7faea4003c10 comp rx=0 tx=0).stop 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 shutdown_connections 2024-04-25T18:55:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fae980a3e40 0x7fae980a6300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faeb0112f40 0x7faeb019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7faeb0074c10 0x7faeb01a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 --2- 172.21.15.63:0/3940780029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faeb00742f0 0x7faeb01a4940 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.509+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 >> 172.21.15.63:0/3940780029 conn(0x7faeb00fda00 msgr2=0x7faeb00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.510+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 shutdown_connections 2024-04-25T18:55:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:38.510+0000 7faeb8600640 1 -- 172.21.15.63:0/3940780029 wait complete. 2024-04-25T18:55:38.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:38 smithi114 ceph-mon[32988]: pgmap v282: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 819 KiB/s wr, 42 op/s 2024-04-25T18:55:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:38 smithi063 ceph-mon[28006]: pgmap v282: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 819 KiB/s wr, 42 op/s 2024-04-25T18:55:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:38 smithi070 ceph-mon[32820]: pgmap v282: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 819 KiB/s wr, 42 op/s 2024-04-25T18:55:39.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 310015 3.91 MB/sec execute 338 sec latency 39.458 ms 2024-04-25T18:55:40.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 -- 172.21.15.63:0/3234399076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 msgr2=0x7fe328105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:40.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3234399076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328105280 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe3180099c0 tx=0x7fe318009cd0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 -- 172.21.15.63:0/3234399076 shutdown_connections 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3234399076 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328105280 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3234399076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328109f80 0x7fe328111b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3234399076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328105850 0x7fe3281098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 -- 172.21.15.63:0/3234399076 >> 172.21.15.63:0/3234399076 conn(0x7fe328100750 msgr2=0x7fe328102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 -- 172.21.15.63:0/3234399076 shutdown_connections 2024-04-25T18:55:40.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.434+0000 7fe32ee00700 1 -- 172.21.15.63:0/3234399076 wait complete. 2024-04-25T18:55:40.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.436+0000 7fe32ee00700 1 Processor -- start 2024-04-25T18:55:40.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.436+0000 7fe32ee00700 1 -- start start 2024-04-25T18:55:40.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:40.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328105850 0x7fe32810eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:40.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe328114e80 con 0x7fe328105850 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe328114d00 con 0x7fe328109f80 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ee00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe328115000 con 0x7fe328104ea0 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe327e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49182/0 (socket says 172.21.15.63:49182) 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe327e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58032/0 (socket says 172.21.15.63:58032) 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 -- 172.21.15.63:0/3568004995 learned_addr learned my addr 172.21.15.63:0/3568004995 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe327400700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328105850 0x7fe32810eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 -- 172.21.15.63:0/3568004995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 msgr2=0x7fe328111a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 -- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328105850 msgr2=0x7fe32810eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328105850 0x7fe32810eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe32ce00700 1 -- 172.21.15.63:0/3568004995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3281abbc0 con 0x7fe328109f80 2024-04-25T18:55:40.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.437+0000 7fe327e00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe32ce00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe320007ee0 tx=0x7fe32000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32000dd20 con 0x7fe328109f80 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe320004020 con 0x7fe328109f80 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe320015660 con 0x7fe328109f80 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3281abd90 con 0x7fe328109f80 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.438+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3281ac270 con 0x7fe328109f80 2024-04-25T18:55:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.439+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe328106380 con 0x7fe328109f80 2024-04-25T18:55:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.441+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe3200041d0 con 0x7fe328109f80 2024-04-25T18:55:40.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.443+0000 7fe324c00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 0x7fe30c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:40.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.443+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7fe3200d0130 con 0x7fe328109f80 2024-04-25T18:55:40.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.443+0000 7fe327e00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 0x7fe30c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:40.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.444+0000 7fe327e00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 0x7fe30c0a62f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe318003760 tx=0x7fe318005fb0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:40.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.447+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe320010070 con 0x7fe328109f80 2024-04-25T18:55:40.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311043 3.91 MB/sec execute 339 sec latency 28.654 ms 2024-04-25T18:55:40.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:40 smithi114 ceph-mon[32988]: pgmap v283: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-25T18:55:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:40 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:40 smithi063 ceph-mon[28006]: pgmap v283: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-25T18:55:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:40 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:40 smithi070 ceph-mon[32820]: pgmap v283: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-25T18:55:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:40 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.727+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe328110010 con 0x7fe328109f80 2024-04-25T18:55:40.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.729+0000 7fe324c00700 1 -- 172.21.15.63:0/3568004995 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11872 (secure 0 0 0) 0x7fe32009c730 con 0x7fe328109f80 2024-04-25T18:55:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:55:40.730 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:55:39.000529+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:55:40.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 msgr2=0x7fe30c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:40.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 0x7fe30c0a62f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe318003760 tx=0x7fe318005fb0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 msgr2=0x7fe32810f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe320007ee0 tx=0x7fe32000d3b0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 shutdown_connections 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe30c0a3e30 0x7fe30c0a62f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe328104ea0 0x7fe328111a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe328109f80 0x7fe32810f120 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 --2- 172.21.15.63:0/3568004995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe328105850 0x7fe32810eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 >> 172.21.15.63:0/3568004995 conn(0x7fe328100750 msgr2=0x7fe3281019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 shutdown_connections 2024-04-25T18:55:40.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:40.732+0000 7fe32ee00700 1 -- 172.21.15.63:0/3568004995 wait complete. 2024-04-25T18:55:40.735 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 35 2024-04-25T18:55:41.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:41 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3568004995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:41.493 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311068 3.90 MB/sec execute 340 sec latency 716.005 ms 2024-04-25T18:55:41.686 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 84.8 --await-for 42.0' 2024-04-25T18:55:41.687 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 84.8 --await-for 42.0 2024-04-25T18:55:41.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:41 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3568004995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:41 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3568004995' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:55:42.171 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:55:42.494 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311068 3.88 MB/sec execute 341 sec latency 1262.698 ms 2024-04-25T18:55:42.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:42 smithi063 ceph-mon[28006]: pgmap v284: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:42 smithi070 ceph-mon[32820]: pgmap v284: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:42.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:42 smithi114 ceph-mon[32988]: pgmap v284: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:43.494 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311068 3.87 MB/sec execute 342 sec latency 2262.830 ms 2024-04-25T18:55:44.494 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311068 3.86 MB/sec execute 343 sec latency 3262.957 ms 2024-04-25T18:55:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 -- 172.21.15.63:0/2224484060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4804108a60 msgr2=0x7f4804108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 --2- 172.21.15.63:0/2224484060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4804108a60 0x7f4804108e40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009990 tx=0x7f47f4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 -- 172.21.15.63:0/2224484060 shutdown_connections 2024-04-25T18:55:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 --2- 172.21.15.63:0/2224484060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406b100 0x7f480410f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 --2- 172.21.15.63:0/2224484060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4804108a60 0x7f4804108e40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 --2- 172.21.15.63:0/2224484060 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f480406a740 0x7f480406abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.599+0000 7f480ae00700 1 -- 172.21.15.63:0/2224484060 >> 172.21.15.63:0/2224484060 conn(0x7f48040788e0 msgr2=0x7f4804078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.600+0000 7f480ae00700 1 -- 172.21.15.63:0/2224484060 shutdown_connections 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.600+0000 7f480ae00700 1 -- 172.21.15.63:0/2224484060 wait complete. 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 Processor -- start 2024-04-25T18:55:44.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 -- start start 2024-04-25T18:55:44.603 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:44 smithi063 ceph-mon[28006]: pgmap v285: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 0x7f480419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4804108a60 0x7f480419fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4804112c40 con 0x7f480406a740 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4804112ac0 con 0x7f480406b100 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.601+0000 7f480ae00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4804112dc0 con 0x7f4804108a60 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 0x7f480419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35662/0 (socket says 172.21.15.63:35662) 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803e00700 1 -- 172.21.15.63:0/2878452070 learned_addr learned my addr 172.21.15.63:0/2878452070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:44.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4808e00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4804108a60 0x7f480419fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 -- 172.21.15.63:0/2878452070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4804108a60 msgr2=0x7f480419fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4804108a60 0x7f480419fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 -- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 msgr2=0x7f480419b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803400700 1 -- 172.21.15.63:0/2878452070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48041a0250 con 0x7f480406b100 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.602+0000 7f4803e00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:44.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.603+0000 7f4803400700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 0x7f480419b780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e9f0 tx=0x7f47f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.603+0000 7f4800c00700 1 -- 172.21.15.63:0/2878452070 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f000c9c0 con 0x7f480406b100 2024-04-25T18:55:44.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.603+0000 7f4800c00700 1 -- 172.21.15.63:0/2878452070 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f000cb60 con 0x7f480406b100 2024-04-25T18:55:44.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.603+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48041a0540 con 0x7f480406b100 2024-04-25T18:55:44.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.603+0000 7f4800c00700 1 -- 172.21.15.63:0/2878452070 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0018680 con 0x7f480406b100 2024-04-25T18:55:44.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.604+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4804109800 con 0x7f480406b100 2024-04-25T18:55:44.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.608+0000 7f4800c00700 1 -- 172.21.15.63:0/2878452070 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47f0010430 con 0x7f480406b100 2024-04-25T18:55:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.609+0000 7f4800c00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 0x7f47e80a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.609+0000 7f4800c00700 1 -- 172.21.15.63:0/2878452070 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f47f00d0ab0 con 0x7f480406b100 2024-04-25T18:55:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.610+0000 7f4803e00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 0x7f47e80a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.610+0000 7f48156aeb80 1 Processor -- start 2024-04-25T18:55:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f4803e00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 0x7f47e80a6390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006010 tx=0x7f47f400b280 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 -- start start 2024-04-25T18:55:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fb9ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55981faee020 con 0x55981fba6180 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55981fa948c0 con 0x55981fb9b2d0 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f48156aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55981fae9fe0 con 0x55981fb9a9b0 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49196/0 (socket says 172.21.15.63:49196) 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f4803400700 1 -- 172.21.15.63:0/415850266 learned_addr learned my addr 172.21.15.63:0/415850266 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:44.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.611+0000 7f4808e00700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803e00700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fb9ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 -- 172.21.15.63:0/415850266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 msgr2=0x55981fb9ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fb9ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 -- 172.21.15.63:0/415850266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 msgr2=0x55981fba8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 -- 172.21.15.63:0/415850266 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981fa8a810 con 0x55981fb9b2d0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803e00700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fb9ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f4803400700 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f47f0098e70 tx=0x7f47f009d190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d4571045eea5541 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f47f9600700 1 -- 172.21.15.63:0/415850266 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f00a0dc0 con 0x55981fb9b2d0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f47f9600700 1 -- 172.21.15.63:0/415850266 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f00a5680 con 0x55981fb9b2d0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.612+0000 7f47f9600700 1 -- 172.21.15.63:0/415850266 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f00bbe30 con 0x55981fb9b2d0 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 -- 172.21.15.63:0/415850266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 msgr2=0x55981fba5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f47f0098e70 tx=0x7f47f009d190 comp rx=0 tx=0).stop 2024-04-25T18:55:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 -- 172.21.15.63:0/415850266 shutdown_connections 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fb9ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 --2- 172.21.15.63:0/415850266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba5c40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 -- 172.21.15.63:0/415850266 >> 172.21.15.63:0/415850266 conn(0x55981fac9730 msgr2=0x55981fb96e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 -- 172.21.15.63:0/415850266 shutdown_connections 2024-04-25T18:55:44.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.613+0000 7f48156aeb80 1 -- 172.21.15.63:0/415850266 wait complete. 2024-04-25T18:55:44.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.614+0000 7f48156aeb80 1 -- start start 2024-04-25T18:55:44.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.614+0000 7f48156aeb80 1 Processor -- start 2024-04-25T18:55:44.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55981fba8be0 con 0x55981fba6180 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55981faa79f0 con 0x55981fb9b2d0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55981fba8d60 con 0x55981fb9a9b0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f48156aeb80 10 client.0 _ll_register_callbacks cb 0x55981fa868b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49206/0 (socket says 172.21.15.63:49206) 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 -- 172.21.15.63:0/3585797437 learned_addr learned my addr 172.21.15.63:0/3585797437 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4808e00700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803e00700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 -- 172.21.15.63:0/3585797437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 msgr2=0x55981fba5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 -- 172.21.15.63:0/3585797437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 msgr2=0x55981fba3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.615+0000 7f4803400700 1 -- 172.21.15.63:0/3585797437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981fa3ba60 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f4803400700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e9c0 tx=0x7f47f00a0e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e59fdfab2faee61a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f47ee000700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0080060 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f47f00a51f0 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f00a5680 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f47f00a5430 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f48156aeb80 10 client.26315 fetch_fsmap learned FSMap version 36 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.616+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55981fa84c80 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f00a5680 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f47f009b8d0 con 0x55981fb9b2d0 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 10 client.26315 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 10 client.26315 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:44.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 20 client.26315 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:44.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fbb5750 0x55981fbb7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 4 client.26315 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:44.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55981fafb810 con 0x55981fbb5750 2024-04-25T18:55:44.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.617+0000 7f4803e00700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fbb5750 0x55981fbb7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.618+0000 7f4803e00700 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fbb5750 0x55981fbb7b40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86579bf5aa240acf server_cookie=d70336b1cbee527e in_seq=0 out_seq=0 2024-04-25T18:55:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.618+0000 7f47ee000700 10 client.26315 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.618+0000 7f47ee000700 1 -- 172.21.15.63:0/3585797437 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55981fafb810 con 0x55981fbb5750 2024-04-25T18:55:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.618+0000 7f47ee000700 10 client.26315 handle_command_reply: tid=0 2024-04-25T18:55:44.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 client.26315 shutdown 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 msgr2=0x55981fba2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e9c0 tx=0x7f47f00a0e70 comp rx=0 tx=0).stop 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 shutdown_connections 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fbb5750 0x55981fbb7b40 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fba3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3585797437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba2c40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 >> 172.21.15.63:0/3585797437 conn(0x55981fac9730 msgr2=0x55981fb96a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f47ee000700 0 client.26315 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 shutdown_connections 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 1 -- 172.21.15.63:0/3585797437 wait complete. 2024-04-25T18:55:44.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.619+0000 7f48156aeb80 20 client.26315 trim_cache size 0 max 16384 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.641+0000 7f48156aeb80 1 Processor -- start 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.641+0000 7f48156aeb80 1 -- start start 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.641+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.641+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.641+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fb9fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f48156aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f48041adb20 con 0x55981fba6180 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f48156aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48041ab2f0 con 0x55981fb9b2d0 2024-04-25T18:55:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f48156aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48041a9160 con 0x55981fb9a9b0 2024-04-25T18:55:44.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4808e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fb9fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:49218/0 (socket says 172.21.15.63:49218) 2024-04-25T18:55:44.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 -- 172.21.15.63:0/2789705719 learned_addr learned my addr 172.21.15.63:0/2789705719 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:44.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803e00700 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 -- 172.21.15.63:0/2789705719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 msgr2=0x55981fba32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 -- 172.21.15.63:0/2789705719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 msgr2=0x55981fb9fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fb9fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.642+0000 7f4803400700 1 -- 172.21.15.63:0/2789705719 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55981fba11b0 con 0x55981fb9b2d0 2024-04-25T18:55:44.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.643+0000 7f4803400700 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47f009d070 tx=0x7f47f0009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47cd53ca533be6a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.643+0000 7f47f8c00700 1 -- 172.21.15.63:0/2789705719 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0080070 con 0x55981fb9b2d0 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.643+0000 7f47f8c00700 1 -- 172.21.15.63:0/2789705719 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f00a5680 con 0x55981fb9b2d0 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.643+0000 7f47f8c00700 1 -- 172.21.15.63:0/2789705719 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f00873f0 con 0x55981fb9b2d0 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- 172.21.15.63:0/2789705719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 msgr2=0x55981fba4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47f009d070 tx=0x7f47f0009790 comp rx=0 tx=0).stop 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- 172.21.15.63:0/2789705719 shutdown_connections 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fb9a9b0 0x55981fba32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fba6180 0x55981fb9fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 --2- 172.21.15.63:0/2789705719 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fba4b80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- 172.21.15.63:0/2789705719 >> 172.21.15.63:0/2789705719 conn(0x55981fac9730 msgr2=0x55981fba4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- 172.21.15.63:0/2789705719 shutdown_connections 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- 172.21.15.63:0/2789705719 wait complete. 2024-04-25T18:55:44.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 -- start start 2024-04-25T18:55:44.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.644+0000 7f48156aeb80 1 Processor -- start 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fb9a9b0 0x55981fbbabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fbb8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f48041a97b0 con 0x55981fb9a9b0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f48041aac20 con 0x55981fb9b2d0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f48041a9e80 con 0x55981fba6180 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f48156aeb80 10 client.0 _ll_register_callbacks cb 0x55981fa868b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f4808e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f4803e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fb9a9b0 0x55981fbbabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f4803400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fbb8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f4808e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58082/0 (socket says 172.21.15.63:58082) 2024-04-25T18:55:44.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.645+0000 7f4808e00700 1 -- 172.21.15.63:0/3477730328 learned_addr learned my addr 172.21.15.63:0/3477730328 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 -- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 msgr2=0x55981fbb8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fbb8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 -- 172.21.15.63:0/3477730328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fb9a9b0 msgr2=0x55981fbbabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fb9a9b0 0x55981fbbabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 -- 172.21.15.63:0/3477730328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55981fc1b940 con 0x55981fba6180 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4803400700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fbb8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f4808e00700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47fc01e5e0 tx=0x7f47fc009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2986e1ef069b49fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f47ee000700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc026070 con 0x55981fba6180 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f00a5680 con 0x55981fba6180 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.646+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f47fc003c00 con 0x55981fba6180 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f47fc032050 con 0x55981fba6180 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 10 client.26344 fetch_fsmap learned FSMap version 36 2024-04-25T18:55:44.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55981fc1ca80 con 0x55981fba6180 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47f00a5680 con 0x55981fba6180 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 36) ==== 8354+0+0 (secure 0 0 0) 0x7f47fc021870 con 0x55981fba6180 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 10 client.26344 fetch_fsmap finished waiting for FSMap version 36 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 10 client.26344 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 20 client.26344 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fc1d290 0x55981fc1f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 4 client.26344 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.8, "await_for": 42.0}] 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.647+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 84.8, "await_for": 42.0}) -- 0x55981faf58b0 con 0x55981fc1d290 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.648+0000 7f4803e00700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fc1d290 0x55981fc1f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:44.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.648+0000 7f4803e00700 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fc1d290 0x55981fc1f6b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d2da1df290843d1 server_cookie=c9c44714431f273b in_seq=0 out_seq=0 2024-04-25T18:55:44.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:44.648+0000 7f47ee000700 10 client.26344 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:44 smithi070 ceph-mon[32820]: pgmap v285: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:44 smithi114 ceph-mon[32988]: pgmap v285: 97 pgs: 97 active+clean; 89 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 75 op/s 2024-04-25T18:55:45.494 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.85 MB/sec execute 344 sec latency 4171.204 ms 2024-04-25T18:55:45.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:45 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:45.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:45 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:45.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:45 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:46.494 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.84 MB/sec execute 345 sec latency 1092.119 ms 2024-04-25T18:55:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:46 smithi063 ceph-mon[28006]: pgmap v286: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-04-25T18:55:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:46 smithi070 ceph-mon[32820]: pgmap v286: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-04-25T18:55:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:46 smithi114 ceph-mon[32988]: pgmap v286: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 97 op/s 2024-04-25T18:55:47.495 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.83 MB/sec execute 346 sec latency 2092.297 ms 2024-04-25T18:55:48.495 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.82 MB/sec execute 347 sec latency 3092.442 ms 2024-04-25T18:55:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:48 smithi063 ceph-mon[28006]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 668 KiB/s wr, 79 op/s 2024-04-25T18:55:48.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:48 smithi070 ceph-mon[32820]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 668 KiB/s wr, 79 op/s 2024-04-25T18:55:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:48 smithi114 ceph-mon[32988]: pgmap v287: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 668 KiB/s wr, 79 op/s 2024-04-25T18:55:49.495 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.81 MB/sec execute 348 sec latency 4092.646 ms 2024-04-25T18:55:50.495 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.80 MB/sec execute 349 sec latency 5092.757 ms 2024-04-25T18:55:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:50 smithi070 ceph-mon[32820]: pgmap v288: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 90 op/s 2024-04-25T18:55:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:50 smithi114 ceph-mon[32988]: pgmap v288: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 90 op/s 2024-04-25T18:55:50.825 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:50 smithi063 ceph-mon[28006]: pgmap v288: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 784 KiB/s wr, 90 op/s 2024-04-25T18:55:51.495 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.78 MB/sec execute 350 sec latency 6092.882 ms 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 -- 172.21.15.63:0/1531638911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d0074380 msgr2=0x7f13d0074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 --2- 172.21.15.63:0/1531638911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d0074380 0x7f13d0074760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009860 tx=0x7f13c002a3d0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 -- 172.21.15.63:0/1531638911 shutdown_connections 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 --2- 172.21.15.63:0/1531638911 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d010e590 0x7f13d01109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 --2- 172.21.15.63:0/1531638911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074ca0 0x7f13d010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 --2- 172.21.15.63:0/1531638911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d0074380 0x7f13d0074760 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 -- 172.21.15.63:0/1531638911 >> 172.21.15.63:0/1531638911 conn(0x7f13d00fd8a0 msgr2=0x7f13d00ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 -- 172.21.15.63:0/1531638911 shutdown_connections 2024-04-25T18:55:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.651+0000 7f13d6600640 1 -- 172.21.15.63:0/1531638911 wait complete. 2024-04-25T18:55:51.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.652+0000 7f13d6600640 1 Processor -- start 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.652+0000 7f13d6600640 1 -- start start 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 0x7f13d01a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d010e590 0x7f13d019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f13d0114220 con 0x7f13d0074380 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01140a0 con 0x7f13d010e590 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d6600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01143a0 con 0x7f13d0074ca0 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d5200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 0x7f13d01a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d5200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 0x7f13d01a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60944/0 (socket says 172.21.15.63:60944) 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:32954/0 (socket says 172.21.15.63:32954) 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d010e590 0x7f13d019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.653+0000 7f13d5200640 1 -- 172.21.15.63:0/2693224336 learned_addr learned my addr 172.21.15.63:0/2693224336 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 -- 172.21.15.63:0/2693224336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d010e590 msgr2=0x7f13d019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d010e590 0x7f13d019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 -- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 msgr2=0x7f13d01a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 0x7f13d01a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 -- 172.21.15.63:0/2693224336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d019ad40 con 0x7f13d0074ca0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13c7e00640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f13b800e790 tx=0x7f13b800ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13ce000640 1 -- 172.21.15.63:0/2693224336 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b800eea0 con 0x7f13d0074ca0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13ce000640 1 -- 172.21.15.63:0/2693224336 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b800ccc0 con 0x7f13d0074ca0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d019b030 con 0x7f13d0074ca0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.654+0000 7f13ce000640 1 -- 172.21.15.63:0/2693224336 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b8018680 con 0x7f13d0074ca0 2024-04-25T18:55:51.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.655+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d01a72d0 con 0x7f13d0074ca0 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.657+0000 7f13daa26740 1 Processor -- start 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 -- start start 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13ce000640 1 -- 172.21.15.63:0/2693224336 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f13b8010430 con 0x7f13d0074ca0 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 0x5594cf1ac650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1aef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5594cf107900 con 0x5594cf1acb90 2024-04-25T18:55:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594cf0adef0 con 0x5594cf1a19b0 2024-04-25T18:55:51.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13daa26740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5594cf1038c0 con 0x5594cf1a22d0 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13d5200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 0x5594cf1ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1aef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41242/0 (socket says 172.21.15.63:41242) 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.658+0000 7f13d5c00640 1 -- 172.21.15.63:0/472023236 learned_addr learned my addr 172.21.15.63:0/472023236 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 -- 172.21.15.63:0/472023236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 msgr2=0x5594cf1a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 -- 172.21.15.63:0/472023236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 msgr2=0x5594cf1aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1aef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 -- 172.21.15.63:0/472023236 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594cee1a210 con 0x5594cf1a22d0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13d5200640 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 0x5594cf1ac650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009830 tx=0x7f13c0032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7802d82ca573475b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.659+0000 7f13c6000640 1 -- 172.21.15.63:0/472023236 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0004050 con 0x5594cf1a22d0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13ce000640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 0x7f13b40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13c6000640 1 -- 172.21.15.63:0/472023236 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c0030e30 con 0x5594cf1a22d0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13c6000640 1 -- 172.21.15.63:0/472023236 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0005590 con 0x5594cf1a22d0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13ce000640 1 -- 172.21.15.63:0/2693224336 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f13b80d09d0 con 0x7f13d0074ca0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13d5c00640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 0x7f13b40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13daa26740 1 -- 172.21.15.63:0/472023236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 msgr2=0x5594cf1ac650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.660+0000 7f13daa26740 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 0x5594cf1ac650 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009830 tx=0x7f13c0032f90 comp rx=0 tx=0).stop 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 -- 172.21.15.63:0/472023236 shutdown_connections 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1aef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a22d0 0x5594cf1ac650 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 --2- 172.21.15.63:0/472023236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.663 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 -- 172.21.15.63:0/472023236 >> 172.21.15.63:0/472023236 conn(0x5594cf19dec0 msgr2=0x5594cf19e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13d5c00640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 0x7f13b40a61d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13c8011630 tx=0x7f13c8009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 -- 172.21.15.63:0/472023236 shutdown_connections 2024-04-25T18:55:51.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 -- 172.21.15.63:0/472023236 wait complete. 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 -- start start 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.661+0000 7f13daa26740 1 Processor -- start 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1acb90 0x5594cf1aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5594cf1af760 con 0x5594cf1a22d0 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5594cf0c1310 con 0x5594cf1a19b0 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5594cf1af8e0 con 0x5594cf1acb90 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13daa26740 10 client.0 _ll_register_callbacks cb 0x5594cf08fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:51.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13d5200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60954/0 (socket says 172.21.15.63:60954) 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13c7e00640 1 -- 172.21.15.63:0/1739701335 learned_addr learned my addr 172.21.15.63:0/1739701335 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.662+0000 7f13d5c00640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1acb90 0x5594cf1aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 -- 172.21.15.63:0/1739701335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1acb90 msgr2=0x5594cf1aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1acb90 0x5594cf1aa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 -- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 msgr2=0x5594cf1a9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 -- 172.21.15.63:0/1739701335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594ceee6620 con 0x5594cf1a22d0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13c7e00640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f13b8005fd0 tx=0x7f13b800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2673b999d56ddd9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13bea00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b809c340 con 0x5594cf1a22d0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f13b80872d0 con 0x5594cf1a22d0 2024-04-25T18:55:51.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.663+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13c0030e30 con 0x5594cf1a22d0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f13b80ce570 con 0x5594cf1a22d0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 10 client.16152 fetch_fsmap learned FSMap version 37 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5594cf1bba20 con 0x5594cf1a22d0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13c0030e30 con 0x5594cf1a22d0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f13b80bbaf0 con 0x5594cf1a22d0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 10 client.16152 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 10 client.16152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 20 client.16152 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf1bcee0 0x5594cf1bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 4 client.16152 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.664+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594cf11b8b0 con 0x5594cf1bcee0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.665+0000 7f13d5200640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf1bcee0 0x5594cf1bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.665+0000 7f13d5200640 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf1bcee0 0x5594cf1bf330 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d562524ffe8e267e server_cookie=6c62e8dd71c25ff3 in_seq=0 out_seq=0 2024-04-25T18:55:51.669 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.665+0000 7f13bea00640 10 client.16152 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.666+0000 7f13bea00640 1 -- 172.21.15.63:0/1739701335 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5594cf11b8b0 con 0x5594cf1bcee0 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.666+0000 7f13bea00640 10 client.16152 handle_command_reply: tid=0 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.666+0000 7f13daa26740 1 client.16152 shutdown 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.666+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 msgr2=0x5594cf1aa060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.666+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f13b8005fd0 tx=0x7f13b800cf30 comp rx=0 tx=0).stop 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 shutdown_connections 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf1bcee0 0x5594cf1bf330 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1acb90 0x5594cf1aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1a22d0 0x5594cf1aa060 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 --2- 172.21.15.63:0/1739701335 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a19b0 0x5594cf1a9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13bea00640 0 client.16152 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:51.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 >> 172.21.15.63:0/1739701335 conn(0x5594cf19dec0 msgr2=0x5594cf1a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 shutdown_connections 2024-04-25T18:55:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 1 -- 172.21.15.63:0/1739701335 wait complete. 2024-04-25T18:55:51.672 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.667+0000 7f13daa26740 20 client.16152 trim_cache size 0 max 16384 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.698+0000 7f13daa26740 1 Processor -- start 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.698+0000 7f13daa26740 1 -- start start 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01b2690 con 0x5594cf1acb90 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01afed0 con 0x5594cf1a22d0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13daa26740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01add40 con 0x5594cf1a19b0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60964/0 (socket says 172.21.15.63:60964) 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13d5c00640 1 -- 172.21.15.63:0/525261169 learned_addr learned my addr 172.21.15.63:0/525261169 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13c7e00640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.699+0000 7f13d5200640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 -- 172.21.15.63:0/525261169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 msgr2=0x5594cf1b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 -- 172.21.15.63:0/525261169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 msgr2=0x5594cf1c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 -- 172.21.15.63:0/525261169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594cf1a4960 con 0x5594cf1a22d0 2024-04-25T18:55:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13d5200640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13d5c00640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c7e00640 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1c4c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13b809c970 tx=0x7f13b80d0d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a7a48579a7ecf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.700+0000 7f13c5600640 1 -- 172.21.15.63:0/525261169 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b809c9e0 con 0x5594cf1a22d0 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13c5600640 1 -- 172.21.15.63:0/525261169 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b80d8050 con 0x5594cf1a22d0 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13c5600640 1 -- 172.21.15.63:0/525261169 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b809be30 con 0x5594cf1a22d0 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 -- 172.21.15.63:0/525261169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 msgr2=0x5594cf1c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1c4c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f13b809c970 tx=0x7f13b80d0d50 comp rx=0 tx=0).stop 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 -- 172.21.15.63:0/525261169 shutdown_connections 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1c4c40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 --2- 172.21.15.63:0/525261169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 -- 172.21.15.63:0/525261169 >> 172.21.15.63:0/525261169 conn(0x5594cf19dec0 msgr2=0x5594cf1a4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 -- 172.21.15.63:0/525261169 shutdown_connections 2024-04-25T18:55:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.701+0000 7f13daa26740 1 -- 172.21.15.63:0/525261169 wait complete. 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 -- start start 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 Processor -- start 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1b2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01ae390 con 0x5594cf1acb90 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01af800 con 0x5594cf1a22d0 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f13d01aea60 con 0x5594cf1a19b0 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.702+0000 7f13daa26740 10 client.0 _ll_register_callbacks cb 0x7f13b8009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13d5c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.710 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41276/0 (socket says 172.21.15.63:41276) 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 -- 172.21.15.63:0/703460141 learned_addr learned my addr 172.21.15.63:0/703460141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13d5200640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 -- 172.21.15.63:0/703460141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 msgr2=0x5594cf1b4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 -- 172.21.15.63:0/703460141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 msgr2=0x5594cf1b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 -- 172.21.15.63:0/703460141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594cf1b2f80 con 0x5594cf1a22d0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13c7e00640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f13b809ac90 tx=0x7f13b80bbd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64315520b4d3ebe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.703+0000 7f13bea00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b80a0ce0 con 0x5594cf1a22d0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 8587+0+0 (secure 0 0 0) 0x7f13b8010930 con 0x5594cf1a22d0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13b80d8050 con 0x5594cf1a22d0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f13b809c9e0 con 0x5594cf1a22d0 2024-04-25T18:55:51.711 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 10 client.26339 fetch_fsmap learned FSMap version 37 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5594cf2234a0 con 0x5594cf1a22d0 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13b80d8050 con 0x5594cf1a22d0 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 37) ==== 8350+0+0 (secure 0 0 0) 0x7f13b809b8a0 con 0x5594cf1a22d0 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 10 client.26339 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 10 client.26339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 20 client.26339 populate_metadata read hostname 'smithi063' 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf223910 0x5594cf225d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 4 client.26339 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.704+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5594cf115320 con 0x5594cf223910 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.705+0000 7f13d5200640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf223910 0x5594cf225d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.705+0000 7f13d5200640 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf223910 0x5594cf225d30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb7b9dfbd3976287 server_cookie=de539c470d2737b6 in_seq=0 out_seq=0 2024-04-25T18:55:51.712 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:51.705+0000 7f13bea00640 10 client.26339 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:52.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.77 MB/sec execute 351 sec latency 7093.426 ms 2024-04-25T18:55:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:52 smithi070 ceph-mon[32820]: pgmap v289: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:52.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:52.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:52 smithi114 ceph-mon[32988]: pgmap v289: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:52.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:52 smithi063 ceph-mon[28006]: pgmap v289: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:53.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.76 MB/sec execute 352 sec latency 8093.587 ms 2024-04-25T18:55:54.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.75 MB/sec execute 353 sec latency 9093.791 ms 2024-04-25T18:55:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:54 smithi070 ceph-mon[32820]: pgmap v290: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:54 smithi114 ceph-mon[32988]: pgmap v290: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:54 smithi063 ceph-mon[28006]: pgmap v290: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 246 KiB/s wr, 33 op/s 2024-04-25T18:55:55.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.74 MB/sec execute 354 sec latency 10093.883 ms 2024-04-25T18:55:56.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.73 MB/sec execute 355 sec latency 11093.965 ms 2024-04-25T18:55:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:56 smithi070 ceph-mon[32820]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 283 KiB/s wr, 45 op/s 2024-04-25T18:55:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:56 smithi114 ceph-mon[32988]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 283 KiB/s wr, 45 op/s 2024-04-25T18:55:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:56 smithi063 ceph-mon[28006]: pgmap v291: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 283 KiB/s wr, 45 op/s 2024-04-25T18:55:57.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.72 MB/sec execute 356 sec latency 12094.084 ms 2024-04-25T18:55:57.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:57 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:57.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:57 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:57 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:57.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:57 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:57.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:57 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:55:57.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:57 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:55:58.496 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.71 MB/sec execute 357 sec latency 13094.229 ms 2024-04-25T18:55:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:58 smithi070 ceph-mon[32820]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 152 KiB/s wr, 22 op/s 2024-04-25T18:55:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:58 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:58 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:55:58 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:55:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:58 smithi114 ceph-mon[32988]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 152 KiB/s wr, 22 op/s 2024-04-25T18:55:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:58 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:58.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:58 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:58.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:55:58 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:55:58.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:58 smithi063 ceph-mon[28006]: pgmap v292: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 152 KiB/s wr, 22 op/s 2024-04-25T18:55:58.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:58 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:55:58.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:58 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:55:58.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:55:58 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:55:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.048+0000 7f13bea00640 1 -- 172.21.15.63:0/703460141 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5594cf115320 con 0x5594cf223910 2024-04-25T18:55:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.048+0000 7f13bea00640 10 client.26339 handle_command_reply: tid=0 2024-04-25T18:55:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f13daa26740 1 client.26339 shutdown 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 msgr2=0x5594cf1b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f13b809ac90 tx=0x7f13b80bbd20 comp rx=0 tx=0).stop 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 shutdown_connections 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5594cf223910 0x5594cf225d30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5594cf1a22d0 0x5594cf1b2240 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5594cf1acb90 0x5594cf1b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 --2- 172.21.15.63:0/703460141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5594cf1a19b0 0x5594cf1b4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.053 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 >> 172.21.15.63:0/703460141 conn(0x5594cf19dec0 msgr2=0x5594cf1a3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.048+0000 7f47ee000700 1 -- 172.21.15.63:0/3477730328 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55981faf58b0 con 0x55981fc1d290 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.048+0000 7f47ee000700 10 client.26344 handle_command_reply: tid=0 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.048+0000 7f48156aeb80 1 client.26344 shutdown 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 msgr2=0x55981fbb9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47fc01e5e0 tx=0x7f47fc009920 comp rx=0 tx=0).stop 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 shutdown_connections 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55981fc1d290 0x55981fc1f6b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55981fba6180 0x55981fbb9540 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55981fb9a9b0 0x55981fbbabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 --2- 172.21.15.63:0/3477730328 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55981fb9b2d0 0x55981fbb8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 >> 172.21.15.63:0/3477730328 conn(0x55981fac9730 msgr2=0x55981fba4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.049+0000 7f47ee000700 0 client.26344 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 shutdown_connections 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f48156aeb80 1 -- 172.21.15.63:0/3477730328 wait complete. 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.050+0000 7f48156aeb80 20 client.26344 trim_cache size 0 max 16384 2024-04-25T18:55:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.051+0000 7f13bea00640 0 client.26339 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:55:59.055 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 38, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 30, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "c393d2af": { 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 30, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 84.8, 2024-04-25T18:55:59.056 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:55:59.057 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:55:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 shutdown_connections 2024-04-25T18:55:59.058 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f13daa26740 1 -- 172.21.15.63:0/703460141 wait complete. 2024-04-25T18:55:59.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 msgr2=0x7f47e80a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 0x7f47e80a6390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f47f4006010 tx=0x7f47f400b280 comp rx=0 tx=0).stop 2024-04-25T18:55:59.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 msgr2=0x7f480419b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.052+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 0x7f480419b780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f47f000e9f0 tx=0x7f47f000ed00 comp rx=0 tx=0).stop 2024-04-25T18:55:59.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 shutdown_connections 2024-04-25T18:55:59.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47e80a3ed0 0x7f47e80a6390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4804108a60 0x7f480419fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f480406a740 0x7f480419b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 --2- 172.21.15.63:0/2878452070 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f480406b100 0x7f480419b780 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 >> 172.21.15.63:0/2878452070 conn(0x7f48040788e0 msgr2=0x7f4804100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:59.059 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.053+0000 7f13daa26740 20 client.26339 trim_cache size 0 max 16384 2024-04-25T18:55:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:55:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:55:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:55:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:55:59.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.054+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 shutdown_connections 2024-04-25T18:55:59.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:55:59.055+0000 7f480ae00700 1 -- 172.21.15.63:0/2878452070 wait complete. 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 msgr2=0x7f13b40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 0x7f13b40a61d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f13c8011630 tx=0x7f13c8009500 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 msgr2=0x7f13d01a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f13b800e790 tx=0x7f13b800ec50 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 shutdown_connections 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f13b40a3d10 0x7f13b40a61d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13d010e590 0x7f13d019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f13d0074ca0 0x7f13d01a0950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 --2- 172.21.15.63:0/2693224336 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13d0074380 0x7f13d01a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:55:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.056+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 >> 172.21.15.63:0/2693224336 conn(0x7f13d00fd8a0 msgr2=0x7f13d00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:55:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.057+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 shutdown_connections 2024-04-25T18:55:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:55:59.057+0000 7f13d6600640 1 -- 172.21.15.63:0/2693224336 wait complete. 2024-04-25T18:55:59.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.70 MB/sec execute 358 sec latency 14094.367 ms 2024-04-25T18:55:59.988 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'c393d2af', quiesce took 22.6 seconds. Will release after: 24.8 2024-04-25T18:56:00.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.69 MB/sec execute 359 sec latency 15094.498 ms 2024-04-25T18:56:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:00 smithi070 ceph-mon[32820]: pgmap v293: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 406 KiB/s wr, 32 op/s 2024-04-25T18:56:00.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:00 smithi114 ceph-mon[32988]: pgmap v293: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 406 KiB/s wr, 32 op/s 2024-04-25T18:56:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:00 smithi063 ceph-mon[28006]: pgmap v293: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 406 KiB/s wr, 32 op/s 2024-04-25T18:56:01.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.68 MB/sec execute 360 sec latency 16094.658 ms 2024-04-25T18:56:02.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.67 MB/sec execute 361 sec latency 17094.792 ms 2024-04-25T18:56:02.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:02 smithi063 ceph-mon[28006]: pgmap v294: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:02.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:02 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:02 smithi070 ceph-mon[32820]: pgmap v294: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:02 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:02 smithi114 ceph-mon[32988]: pgmap v294: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:02 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:03.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.66 MB/sec execute 362 sec latency 18094.887 ms 2024-04-25T18:56:04.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.65 MB/sec execute 363 sec latency 19095.052 ms 2024-04-25T18:56:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:04 smithi070 ceph-mon[32820]: pgmap v295: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:04 smithi114 ceph-mon[32988]: pgmap v295: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:04 smithi063 ceph-mon[28006]: pgmap v295: 97 pgs: 97 active+clean; 93 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 290 KiB/s wr, 21 op/s 2024-04-25T18:56:05.497 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.64 MB/sec execute 364 sec latency 20095.188 ms 2024-04-25T18:56:06.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.63 MB/sec execute 365 sec latency 21095.330 ms 2024-04-25T18:56:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:06 smithi070 ceph-mon[32820]: pgmap v296: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 361 KiB/s wr, 49 op/s 2024-04-25T18:56:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:06 smithi114 ceph-mon[32988]: pgmap v296: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 361 KiB/s wr, 49 op/s 2024-04-25T18:56:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:06 smithi063 ceph-mon[28006]: pgmap v296: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 361 KiB/s wr, 49 op/s 2024-04-25T18:56:07.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.62 MB/sec execute 366 sec latency 22095.492 ms 2024-04-25T18:56:08.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.61 MB/sec execute 367 sec latency 23095.637 ms 2024-04-25T18:56:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:08 smithi070 ceph-mon[32820]: pgmap v297: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 325 KiB/s wr, 37 op/s 2024-04-25T18:56:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:08 smithi114 ceph-mon[32988]: pgmap v297: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 325 KiB/s wr, 37 op/s 2024-04-25T18:56:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:08 smithi063 ceph-mon[28006]: pgmap v297: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 325 KiB/s wr, 37 op/s 2024-04-25T18:56:09.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.60 MB/sec execute 368 sec latency 24095.721 ms 2024-04-25T18:56:10.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.59 MB/sec execute 369 sec latency 25095.814 ms 2024-04-25T18:56:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:10 smithi114 ceph-mon[32988]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 336 KiB/s wr, 41 op/s 2024-04-25T18:56:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:10 smithi114 ceph-mon[32988]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T18:56:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:10 smithi063 ceph-mon[28006]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 336 KiB/s wr, 41 op/s 2024-04-25T18:56:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:10 smithi063 ceph-mon[28006]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T18:56:10.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:10 smithi070 ceph-mon[32820]: pgmap v298: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 336 KiB/s wr, 41 op/s 2024-04-25T18:56:10.956 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:10 smithi070 ceph-mon[32820]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T18:56:11.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.58 MB/sec execute 370 sec latency 26095.912 ms 2024-04-25T18:56:12.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 -- 172.21.15.63:0/427371444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c104bb0 msgr2=0x7f201c108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 --2- 172.21.15.63:0/427371444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c104bb0 0x7f201c108bf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2008009900 tx=0x7f200802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 -- 172.21.15.63:0/427371444 shutdown_connections 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 --2- 172.21.15.63:0/427371444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c104bb0 0x7f201c108bf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 --2- 172.21.15.63:0/427371444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c1092d0 0x7f201c110ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 --2- 172.21.15.63:0/427371444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c104200 0x7f201c1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 -- 172.21.15.63:0/427371444 >> 172.21.15.63:0/427371444 conn(0x7f201c0ffa60 msgr2=0x7f201c101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 -- 172.21.15.63:0/427371444 shutdown_connections 2024-04-25T18:56:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.215+0000 7f2022e00640 1 -- 172.21.15.63:0/427371444 wait complete. 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.216+0000 7f2022e00640 1 Processor -- start 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 -- start start 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c1092d0 0x7f201c1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c19a680 0x7f201c19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1141f0 con 0x7f201c104200 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f201c114070 con 0x7f201c19a680 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2022e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f201c114370 con 0x7f201c1092d0 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2021a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2021a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33046/0 (socket says 172.21.15.63:33046) 2024-04-25T18:56:12.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2021a00640 1 -- 172.21.15.63:0/1852312010 learned_addr learned my addr 172.21.15.63:0/1852312010 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.217+0000 7f2021000640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c1092d0 0x7f201c1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2022400640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c19a680 0x7f201c19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 -- 172.21.15.63:0/1852312010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c1092d0 msgr2=0x7f201c1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c1092d0 0x7f201c1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 -- 172.21.15.63:0/1852312010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c19a680 msgr2=0x7f201c19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c19a680 0x7f201c19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 -- 172.21.15.63:0/1852312010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c19b280 con 0x7f201c104200 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f2021a00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f201000e9f0 tx=0x7f201000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.218+0000 7f201a000640 1 -- 172.21.15.63:0/1852312010 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201000cd80 con 0x7f201c104200 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.219+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c1a7360 con 0x7f201c104200 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.219+0000 7f201a000640 1 -- 172.21.15.63:0/1852312010 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20100108e0 con 0x7f201c104200 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.219+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1a7760 con 0x7f201c104200 2024-04-25T18:56:12.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.219+0000 7f201a000640 1 -- 172.21.15.63:0/1852312010 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2010018920 con 0x7f201c104200 2024-04-25T18:56:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.222+0000 7f20271f4740 1 Processor -- start 2024-04-25T18:56:12.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.222+0000 7f201a000640 1 -- 172.21.15.63:0/1852312010 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2010010430 con 0x7f201c104200 2024-04-25T18:56:12.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.222+0000 7f20271f4740 1 -- start start 2024-04-25T18:56:12.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b31dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b328650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b32af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b90b283900 con 0x55b90b31e2d0 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b90b229ef0 con 0x55b90b328b90 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f20271f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b90b27f8c0 con 0x55b90b31d9b0 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f2021a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b328650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f2022400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b31dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f2021a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b328650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33062/0 (socket says 172.21.15.63:33062) 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f2021a00640 1 -- 172.21.15.63:0/3891819016 learned_addr learned my addr 172.21.15.63:0/3891819016 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.223+0000 7f2021000640 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b32af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 -- 172.21.15.63:0/3891819016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 msgr2=0x55b90b32af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b32af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 -- 172.21.15.63:0/3891819016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 msgr2=0x55b90b328650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b328650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 -- 172.21.15.63:0/3891819016 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b90af96210 con 0x55b90b31d9b0 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f201a000640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 0x7f20040a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2022400640 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b31dd90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2014009f00 tx=0x7f2014011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca456dcfff7f7c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f201a000640 1 -- 172.21.15.63:0/1852312010 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8631+0+0 (secure 0 0 0) 0x7f20100d0b20 con 0x7f201c104200 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f2021a00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 0x7f20040a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.224+0000 7f200ea00640 1 -- 172.21.15.63:0/3891819016 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014004d20 con 0x55b90b31d9b0 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f200ea00640 1 -- 172.21.15.63:0/3891819016 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2014004050 con 0x55b90b31d9b0 2024-04-25T18:56:12.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f200ea00640 1 -- 172.21.15.63:0/3891819016 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014005280 con 0x55b90b31d9b0 2024-04-25T18:56:12.231 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 -- 172.21.15.63:0/3891819016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 msgr2=0x55b90b31dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f2021a00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 0x7f20040a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20100a5190 tx=0x7f20100d4670 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b31dd90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2014009f00 tx=0x7f2014011810 comp rx=0 tx=0).stop 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 -- 172.21.15.63:0/3891819016 shutdown_connections 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b32af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b328650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 --2- 172.21.15.63:0/3891819016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b31dd90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.225+0000 7f20271f4740 1 -- 172.21.15.63:0/3891819016 >> 172.21.15.63:0/3891819016 conn(0x55b90b319ec0 msgr2=0x55b90b31a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.226+0000 7f20271f4740 1 -- 172.21.15.63:0/3891819016 shutdown_connections 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.226+0000 7f20271f4740 1 -- 172.21.15.63:0/3891819016 wait complete. 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.226+0000 7f20271f4740 1 -- start start 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.226+0000 7f20271f4740 1 Processor -- start 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31d9b0 0x55b90b325b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 0x55b90b3265a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b90b32b7c0 con 0x55b90b328b90 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b90b23d310 con 0x55b90b31d9b0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55b90b32b940 con 0x55b90b31e2d0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f20271f4740 10 client.0 _ll_register_callbacks cb 0x55b90b20bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44202/0 (socket says 172.21.15.63:44202) 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 -- 172.21.15.63:0/376577029 learned_addr learned my addr 172.21.15.63:0/376577029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2022400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31d9b0 0x55b90b325b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2022400640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 0x55b90b3265a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 -- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31d9b0 msgr2=0x55b90b325b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31d9b0 0x55b90b325b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 -- 172.21.15.63:0/376577029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 msgr2=0x55b90b3265a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 0x55b90b3265a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2021000640 1 -- 172.21.15.63:0/376577029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b90b062620 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.227+0000 7f2022400640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 0x55b90b3265a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f2021000640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2008033000 tx=0x7f2008032de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf8026196e8ce17e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f2003400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008035040 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8631+0+0 (secure 0 0 0) 0x7f2008004050 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2014004050 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f2008004250 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f20271f4740 10 client.26374 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.228+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b90b337a20 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2014004050 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f20080048f0 con 0x55b90b31e2d0 2024-04-25T18:56:12.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 10 client.26374 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 10 client.26374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 20 client.26374 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b338fa0 0x55b90b33b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 4 client.26374 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.229+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b90b2978b0 con 0x55b90b338fa0 2024-04-25T18:56:12.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.230+0000 7f2022400640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b338fa0 0x55b90b33b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.230+0000 7f2022400640 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b338fa0 0x55b90b33b390 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c06627a5e3d47c2e server_cookie=3278cc96e79d1fb8 in_seq=0 out_seq=0 2024-04-25T18:56:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.230+0000 7f2003400640 10 client.26374 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f2003400640 1 -- 172.21.15.63:0/376577029 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b90b2978b0 con 0x55b90b338fa0 2024-04-25T18:56:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f2003400640 10 client.26374 handle_command_reply: tid=0 2024-04-25T18:56:12.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 client.26374 shutdown 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 msgr2=0x55b90b326060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2008033000 tx=0x7f2008032de0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 shutdown_connections 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b338fa0 0x55b90b33b390 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b328b90 0x55b90b3265a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31e2d0 0x55b90b326060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 --2- 172.21.15.63:0/376577029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31d9b0 0x55b90b325b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f2003400640 0 client.26374 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 >> 172.21.15.63:0/376577029 conn(0x55b90b319ec0 msgr2=0x55b90b31fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 shutdown_connections 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 1 -- 172.21.15.63:0/376577029 wait complete. 2024-04-25T18:56:12.237 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.231+0000 7f20271f4740 20 client.26374 trim_cache size 0 max 16384 2024-04-25T18:56:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 Processor -- start 2024-04-25T18:56:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 -- start start 2024-04-25T18:56:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b3263c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b327a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1b2730 con 0x55b90b31e2d0 2024-04-25T18:56:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1aff70 con 0x55b90b328b90 2024-04-25T18:56:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.248+0000 7f20271f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1adde0 con 0x55b90b31d9b0 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2021000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b3263c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44210/0 (socket says 172.21.15.63:44210) 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 -- 172.21.15.63:0/1707596191 learned_addr learned my addr 172.21.15.63:0/1707596191 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b327a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 -- 172.21.15.63:0/1707596191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 msgr2=0x55b90b327a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b327a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 -- 172.21.15.63:0/1707596191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 msgr2=0x55b90b3263c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b3263c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 -- 172.21.15.63:0/1707596191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b90b33b700 con 0x55b90b31d9b0 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f2022400640 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2014005cc0 tx=0x7f2014009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e5abff996ca08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.249+0000 7f200e000640 1 -- 172.21.15.63:0/1707596191 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014005110 con 0x55b90b31d9b0 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f200e000640 1 -- 172.21.15.63:0/1707596191 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2014005760 con 0x55b90b31d9b0 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f200e000640 1 -- 172.21.15.63:0/1707596191 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014027af0 con 0x55b90b31d9b0 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 -- 172.21.15.63:0/1707596191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 msgr2=0x55b90b334680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2014005cc0 tx=0x7f2014009500 comp rx=0 tx=0).stop 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 -- 172.21.15.63:0/1707596191 shutdown_connections 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b328b90 0x55b90b327a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31e2d0 0x55b90b3263c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 --2- 172.21.15.63:0/1707596191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b31d9b0 0x55b90b334680 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 -- 172.21.15.63:0/1707596191 >> 172.21.15.63:0/1707596191 conn(0x55b90b319ec0 msgr2=0x55b90b327350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 -- 172.21.15.63:0/1707596191 shutdown_connections 2024-04-25T18:56:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.250+0000 7f20271f4740 1 -- 172.21.15.63:0/1707596191 wait complete. 2024-04-25T18:56:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 -- start start 2024-04-25T18:56:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 Processor -- start 2024-04-25T18:56:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 0x55b90b330df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 0x55b90b32e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b328b90 0x55b90b32e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1ae430 con 0x55b90b31d9b0 2024-04-25T18:56:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1af8a0 con 0x55b90b31e2d0 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1aeb00 con 0x55b90b328b90 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.251+0000 7f20271f4740 10 client.0 _ll_register_callbacks cb 0x7f2008009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2022400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 0x55b90b330df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2022400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 0x55b90b330df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33102/0 (socket says 172.21.15.63:33102) 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 0x55b90b32e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2022400640 1 -- 172.21.15.63:0/1172385552 learned_addr learned my addr 172.21.15.63:0/1172385552 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2022400640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b328b90 0x55b90b32e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 -- 172.21.15.63:0/1172385552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b328b90 msgr2=0x55b90b32e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b328b90 0x55b90b32e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 -- 172.21.15.63:0/1172385552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 msgr2=0x55b90b330df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 0x55b90b330df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 -- 172.21.15.63:0/1172385552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b90b32ed60 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2021000640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 0x55b90b32e180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2008004e60 tx=0x7f2008005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e9fd1d162919e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.252+0000 7f2003400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2008035040 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2014005760 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 8631+0+0 (secure 0 0 0) 0x7f2008031ac0 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f200805e050 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f20271f4740 10 client.26357 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b90b39f3c0 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.253+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2014005760 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f2008003800 con 0x55b90b31e2d0 2024-04-25T18:56:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 10 client.26357 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 10 client.26357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 20 client.26357 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b39fb30 0x55b90b3a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 4 client.26357 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b90b291320 con 0x55b90b39fb30 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f2022400640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b39fb30 0x55b90b3a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f2022400640 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b39fb30 0x55b90b3a1f50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58276c1ea7ba7efc server_cookie=d8a6931fdd7355f8 in_seq=0 out_seq=0 2024-04-25T18:56:12.260 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.254+0000 7f2003400640 10 client.26357 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:12.280 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f2003400640 1 -- 172.21.15.63:0/1172385552 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b90b291320 con 0x55b90b39fb30 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f2003400640 10 client.26357 handle_command_reply: tid=0 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 client.26357 shutdown 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 msgr2=0x55b90b32e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 0x55b90b32e180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2008004e60 tx=0x7f2008005bd0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 shutdown_connections 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55b90b39fb30 0x55b90b3a1f50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b90b31e2d0 0x55b90b32e180 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55b90b328b90 0x55b90b32e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 --2- 172.21.15.63:0/1172385552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b90b31d9b0 0x55b90b330df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 >> 172.21.15.63:0/1172385552 conn(0x55b90b319ec0 msgr2=0x55b90b31fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f2003400640 0 client.26357 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 shutdown_connections 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 1 -- 172.21.15.63:0/1172385552 wait complete. 2024-04-25T18:56:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.278+0000 7f20271f4740 20 client.26357 trim_cache size 0 max 16384 2024-04-25T18:56:12.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:56:12.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:56:12.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:56:12.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 msgr2=0x7f20040a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 0x7f20040a61d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f20100a5190 tx=0x7f20100d4670 comp rx=0 tx=0).stop 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 msgr2=0x7f201c1a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f201000e9f0 tx=0x7f201000eeb0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 shutdown_connections 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f20040a3d10 0x7f20040a61d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f201c19a680 0x7f201c19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f201c1092d0 0x7f201c1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 --2- 172.21.15.63:0/1852312010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f201c104200 0x7f201c1a05c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.279+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 >> 172.21.15.63:0/1852312010 conn(0x7f201c0ffa60 msgr2=0x7f201c100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.280+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 shutdown_connections 2024-04-25T18:56:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:12.280+0000 7f2022e00640 1 -- 172.21.15.63:0/1852312010 wait complete. 2024-04-25T18:56:12.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.57 MB/sec execute 371 sec latency 27096.045 ms 2024-04-25T18:56:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:12 smithi114 ceph-mon[32988]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:12 smithi063 ceph-mon[28006]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:12 smithi070 ceph-mon[32820]: pgmap v300: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:13.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.56 MB/sec execute 372 sec latency 28096.187 ms 2024-04-25T18:56:14.498 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.55 MB/sec execute 373 sec latency 29096.332 ms 2024-04-25T18:56:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:14 smithi114 ceph-mon[32988]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:14 smithi063 ceph-mon[28006]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:14 smithi070 ceph-mon[32820]: pgmap v301: 97 pgs: 97 active+clean; 91 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 98 KiB/s wr, 38 op/s 2024-04-25T18:56:15.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.54 MB/sec execute 374 sec latency 30096.476 ms 2024-04-25T18:56:16.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.53 MB/sec execute 375 sec latency 31096.634 ms 2024-04-25T18:56:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:16 smithi114 ceph-mon[32988]: pgmap v302: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:16 smithi063 ceph-mon[28006]: pgmap v302: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:16 smithi070 ceph-mon[32820]: pgmap v302: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:17.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.52 MB/sec execute 376 sec latency 32096.798 ms 2024-04-25T18:56:18.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.51 MB/sec execute 377 sec latency 33096.900 ms 2024-04-25T18:56:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:18 smithi114 ceph-mon[32988]: pgmap v303: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:18 smithi063 ceph-mon[28006]: pgmap v303: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:18 smithi070 ceph-mon[32820]: pgmap v303: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:56:19.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.50 MB/sec execute 378 sec latency 34097.032 ms 2024-04-25T18:56:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:19 smithi114 ceph-mon[32988]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T18:56:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:19 smithi114 ceph-mon[32988]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:19 smithi063 ceph-mon[28006]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T18:56:20.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:19 smithi063 ceph-mon[28006]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:19 smithi070 ceph-mon[32820]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T18:56:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:19 smithi070 ceph-mon[32820]: pgmap v305: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:20.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.50 MB/sec execute 379 sec latency 35097.124 ms 2024-04-25T18:56:21.499 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.49 MB/sec execute 380 sec latency 36097.285 ms 2024-04-25T18:56:22.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.48 MB/sec execute 381 sec latency 37097.419 ms 2024-04-25T18:56:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:22 smithi114 ceph-mon[32988]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:22 smithi063 ceph-mon[28006]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:22 smithi070 ceph-mon[32820]: pgmap v306: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:23.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.47 MB/sec execute 382 sec latency 38097.567 ms 2024-04-25T18:56:24.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.46 MB/sec execute 383 sec latency 39097.720 ms 2024-04-25T18:56:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:24 smithi114 ceph-mon[32988]: pgmap v307: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:24 smithi063 ceph-mon[28006]: pgmap v307: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:24 smithi070 ceph-mon[32820]: pgmap v307: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 26 KiB/s wr, 13 op/s 2024-04-25T18:56:24.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:56:25.305 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:56:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 -- 172.21.15.63:0/3281171015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd670108560 msgr2=0x7fd6701089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 --2- 172.21.15.63:0/3281171015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd670108560 0x7fd6701089e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009890 tx=0x7fd65c02a360 comp rx=0 tx=0).stop 2024-04-25T18:56:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 -- 172.21.15.63:0/3281171015 shutdown_connections 2024-04-25T18:56:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 --2- 172.21.15.63:0/3281171015 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 0x7fd6701153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 --2- 172.21.15.63:0/3281171015 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd670108560 0x7fd6701089e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 --2- 172.21.15.63:0/3281171015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd67010e620 0x7fd67010ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.418+0000 7fd678400640 1 -- 172.21.15.63:0/3281171015 >> 172.21.15.63:0/3281171015 conn(0x7fd6700764e0 msgr2=0x7fd670078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.419+0000 7fd678400640 1 -- 172.21.15.63:0/3281171015 shutdown_connections 2024-04-25T18:56:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.419+0000 7fd678400640 1 -- 172.21.15.63:0/3281171015 wait complete. 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 Processor -- start 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 -- start start 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 0x7fd6701a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701187b0 con 0x7fd670108560 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd670118630 con 0x7fd670108f20 2024-04-25T18:56:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.420+0000 7fd678400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd670118930 con 0x7fd67010e620 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd675a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57750/0 (socket says 172.21.15.63:57750) 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 -- 172.21.15.63:0/1453507023 learned_addr learned my addr 172.21.15.63:0/1453507023 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd675a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34332/0 (socket says 172.21.15.63:34332) 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd675000640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 0x7fd6701a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 -- 172.21.15.63:0/1453507023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 msgr2=0x7fd6701a4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 0x7fd6701a4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 -- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 msgr2=0x7fd6701a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd676400640 1 -- 172.21.15.63:0/1453507023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67019f1d0 con 0x7fd67010e620 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.421+0000 7fd675a00640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd676400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd66800eb10 tx=0x7fd66800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd66e000640 1 -- 172.21.15.63:0/1453507023 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668004b20 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd66e000640 1 -- 172.21.15.63:0/1453507023 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd668004cc0 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67019f4c0 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd66e000640 1 -- 172.21.15.63:0/1453507023 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd668010670 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.422+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6701ab760 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.425+0000 7fd67b1bb740 1 Processor -- start 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.425+0000 7fd66e000640 1 -- 172.21.15.63:0/1453507023 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd668005020 con 0x7fd67010e620 2024-04-25T18:56:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.425+0000 7fd67b1bb740 1 -- start start 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d424197820 0x55d424197c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d4241a2a00 0x55d4241a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d4240fd900 con 0x55d4241a2a00 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4240a3ef0 con 0x55d424198140 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d4240f98c0 con 0x55d424197820 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd675000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d424197820 0x55d424197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd675000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d4241a2a00 0x55d4241a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34352/0 (socket says 172.21.15.63:34352) 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.426+0000 7fd675a00640 1 -- 172.21.15.63:0/3251113347 learned_addr learned my addr 172.21.15.63:0/3251113347 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:25.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 -- 172.21.15.63:0/3251113347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d424197820 msgr2=0x55d424197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d424197820 0x55d424197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 -- 172.21.15.63:0/3251113347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d4241a2a00 msgr2=0x55d4241a4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d4241a2a00 0x55d4241a4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 -- 172.21.15.63:0/3251113347 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d423e10210 con 0x55d424198140 2024-04-25T18:56:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd675a00640 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd6640114f0 tx=0x7fd6640119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2acd663f7c4abb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd66e000640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 0x7fd6580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd662a00640 1 -- 172.21.15.63:0/3251113347 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66401e070 con 0x55d424198140 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.427+0000 7fd66e000640 1 -- 172.21.15.63:0/1453507023 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fd6680d0d50 con 0x7fd67010e620 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd662a00640 1 -- 172.21.15.63:0/3251113347 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd66400fa70 con 0x55d424198140 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd675000640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 0x7fd6580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd662a00640 1 -- 172.21.15.63:0/3251113347 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664019bf0 con 0x55d424198140 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 -- 172.21.15.63:0/3251113347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 msgr2=0x55d4241a24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd6640114f0 tx=0x7fd6640119b0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd675000640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 0x7fd6580a6300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd65c035040 tx=0x7fd65c0056d0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 -- 172.21.15.63:0/3251113347 shutdown_connections 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a24c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d4241a2a00 0x55d4241a4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/3251113347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d424197820 0x55d424197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 -- 172.21.15.63:0/3251113347 >> 172.21.15.63:0/3251113347 conn(0x55d424193d30 msgr2=0x55d424194110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.428+0000 7fd67b1bb740 1 -- 172.21.15.63:0/3251113347 shutdown_connections 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.429+0000 7fd67b1bb740 1 -- 172.21.15.63:0/3251113347 wait complete. 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.429+0000 7fd67b1bb740 1 -- start start 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.429+0000 7fd67b1bb740 1 Processor -- start 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d42419ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d4241a5630 con 0x55d424198140 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55d4240b7310 con 0x55d424197820 2024-04-25T18:56:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55d4241a57b0 con 0x55d4241a2a00 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd67b1bb740 10 client.0 _ll_register_callbacks cb 0x55d424085cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d42419ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34358/0 (socket says 172.21.15.63:34358) 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 -- 172.21.15.63:0/2292113934 learned_addr learned my addr 172.21.15.63:0/2292113934 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd676400640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 -- 172.21.15.63:0/2292113934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 msgr2=0x55d4241a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 -- 172.21.15.63:0/2292113934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 msgr2=0x55d42419ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d42419ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.430+0000 7fd675a00640 1 -- 172.21.15.63:0/2292113934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d423edc620 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd675a00640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd66401c040 tx=0x7fd664009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f4dd57520f8e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd657400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66401e040 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fd664016040 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd66400fa70 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd664031050 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd67b1bb740 10 client.26369 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.431+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d4241b18f0 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd66400fa70 con 0x55d424197820 2024-04-25T18:56:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7fd664005160 con 0x55d424197820 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 10 client.26369 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 10 client.26369 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 20 client.26369 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d4241b2cf0 0x55d4241b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 4 client.26369 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.432+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4241118b0 con 0x55d4241b2cf0 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.433+0000 7fd675000640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d4241b2cf0 0x55d4241b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.433+0000 7fd675000640 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d4241b2cf0 0x55d4241b50e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac8d95905478eb75 server_cookie=8336676b44a7f810 in_seq=0 out_seq=0 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.433+0000 7fd657400640 10 client.26369 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd657400640 1 -- 172.21.15.63:0/2292113934 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d4241118b0 con 0x55d4241b2cf0 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd657400640 10 client.26369 handle_command_reply: tid=0 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 client.26369 shutdown 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 msgr2=0x55d42419f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd66401c040 tx=0x7fd664009790 comp rx=0 tx=0).stop 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 shutdown_connections 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d4241b2cf0 0x55d4241b50e0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d42419ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2292113934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d42419f9f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd657400640 0 client.26369 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 >> 172.21.15.63:0/2292113934 conn(0x55d424193d30 msgr2=0x55d424199a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.434+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 shutdown_connections 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.435+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2292113934 wait complete. 2024-04-25T18:56:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.435+0000 7fd67b1bb740 20 client.26369 trim_cache size 0 max 16384 2024-04-25T18:56:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 Processor -- start 2024-04-25T18:56:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 -- start start 2024-04-25T18:56:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d4241ba8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241bae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b6bb0 con 0x55d424198140 2024-04-25T18:56:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b43f0 con 0x55d424197820 2024-04-25T18:56:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.451+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b2260 con 0x55d4241a2a00 2024-04-25T18:56:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d4241ba8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34362/0 (socket says 172.21.15.63:34362) 2024-04-25T18:56:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 -- 172.21.15.63:0/2210449195 learned_addr learned my addr 172.21.15.63:0/2210449195 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd676400640 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241bae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 -- 172.21.15.63:0/2210449195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 msgr2=0x55d4241bae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241bae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 -- 172.21.15.63:0/2210449195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 msgr2=0x55d4241ba8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d4241ba8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 -- 172.21.15.63:0/2210449195 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d42419df60 con 0x55d424197820 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675000640 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d4241ba8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.452+0000 7fd675a00640 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd6640114f0 tx=0x7fd664005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d66c38835c3cf2e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd662000640 1 -- 172.21.15.63:0/2210449195 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66401e070 con 0x55d424197820 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd662000640 1 -- 172.21.15.63:0/2210449195 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd664009260 con 0x55d424197820 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd662000640 1 -- 172.21.15.63:0/2210449195 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66400f5c0 con 0x55d424197820 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2210449195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 msgr2=0x55d4241ae4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd6640114f0 tx=0x7fd664005970 comp rx=0 tx=0).stop 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2210449195 shutdown_connections 2024-04-25T18:56:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241bae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424198140 0x55d4241ba8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/2210449195 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424197820 0x55d4241ae4f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2210449195 >> 172.21.15.63:0/2210449195 conn(0x55d424193d30 msgr2=0x55d4241a1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2210449195 shutdown_connections 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.453+0000 7fd67b1bb740 1 -- 172.21.15.63:0/2210449195 wait complete. 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.454+0000 7fd67b1bb740 1 -- start start 2024-04-25T18:56:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.454+0000 7fd67b1bb740 1 Processor -- start 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.454+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.454+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd67b1bb740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b28b0 con 0x55d424197820 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b3d20 con 0x55d424198140 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd67b1bb740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd6701b2f80 con 0x55d4241a2a00 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd67b1bb740 10 client.0 _ll_register_callbacks cb 0x7fd664009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34354/0 (socket says 172.21.15.63:34354) 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 -- 172.21.15.63:0/4068385391 learned_addr learned my addr 172.21.15.63:0/4068385391 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675000640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd676400640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 -- 172.21.15.63:0/4068385391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 msgr2=0x55d4241a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 -- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 msgr2=0x55d4241a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675a00640 1 -- 172.21.15.63:0/4068385391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4241a98c0 con 0x55d424197820 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.455+0000 7fd675000640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd675a00640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd66400a010 tx=0x7fd664009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f256325a2c2e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd657400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66401e030 con 0x55d424197820 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fd664005040 con 0x55d424197820 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd664009260 con 0x55d424197820 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd664011310 con 0x55d424197820 2024-04-25T18:56:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd67b1bb740 10 client.16194 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.456+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d424219ec0 con 0x55d424197820 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd664009260 con 0x55d424197820 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7fd664019620 con 0x55d424197820 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 10 client.16194 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 20 client.16194 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d42421a4e0 0x55d42421c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 4 client.16194 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d42410b320 con 0x55d42421a4e0 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.457+0000 7fd675000640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d42421a4e0 0x55d42421c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.458+0000 7fd675000640 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d42421a4e0 0x55d42421c900 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bb0de5ac2ea9e6d server_cookie=9be81354d2fc3f7 in_seq=0 out_seq=0 2024-04-25T18:56:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.458+0000 7fd657400640 10 client.16194 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd657400640 1 -- 172.21.15.63:0/4068385391 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d42410b320 con 0x55d42421a4e0 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd657400640 10 client.16194 handle_command_reply: tid=0 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd67b1bb740 1 client.16194 shutdown 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 msgr2=0x55d4241aaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd66400a010 tx=0x7fd664009800 comp rx=0 tx=0).stop 2024-04-25T18:56:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 shutdown_connections 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.477+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55d42421a4e0 0x55d42421c900 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55d424198140 0x55d4241a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55d4241a2a00 0x55d4241a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 --2- 172.21.15.63:0/4068385391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d424197820 0x55d4241aaaa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 >> 172.21.15.63:0/4068385391 conn(0x55d424193d30 msgr2=0x55d424192600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd657400640 0 client.16194 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 shutdown_connections 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 1 -- 172.21.15.63:0/4068385391 wait complete. 2024-04-25T18:56:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.478+0000 7fd67b1bb740 20 client.16194 trim_cache size 0 max 16384 2024-04-25T18:56:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:56:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:56:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:56:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 msgr2=0x7fd6580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 0x7fd6580a6300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd65c035040 tx=0x7fd65c0056d0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 msgr2=0x7fd67019ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd66800eb10 tx=0x7fd66800c2d0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 shutdown_connections 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd6580a3e40 0x7fd6580a6300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd670108f20 0x7fd6701a4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd67010e620 0x7fd67019ea50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 --2- 172.21.15.63:0/1453507023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd670108560 0x7fd6701a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 >> 172.21.15.63:0/1453507023 conn(0x7fd6700764e0 msgr2=0x7fd670077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 shutdown_connections 2024-04-25T18:56:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:25.479+0000 7fd678400640 1 -- 172.21.15.63:0/1453507023 wait complete. 2024-04-25T18:56:25.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.45 MB/sec execute 384 sec latency 40097.793 ms 2024-04-25T18:56:26.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.44 MB/sec execute 385 sec latency 41097.912 ms 2024-04-25T18:56:26.507 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:26 smithi063 ceph-mon[28006]: pgmap v308: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:26 smithi114 ceph-mon[32988]: pgmap v308: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:26 smithi070 ceph-mon[32820]: pgmap v308: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:27.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.43 MB/sec execute 386 sec latency 42098.018 ms 2024-04-25T18:56:27.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 -- 172.21.15.63:0/3663095214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 msgr2=0x7f3d20108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 --2- 172.21.15.63:0/3663095214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20108e10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d100098f0 tx=0x7f3d10009c00 comp rx=0 tx=0).stop 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 -- 172.21.15.63:0/3663095214 shutdown_connections 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 --2- 172.21.15.63:0/3663095214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d2010f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 --2- 172.21.15.63:0/3663095214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20108e10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 --2- 172.21.15.63:0/3663095214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d2006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 -- 172.21.15.63:0/3663095214 >> 172.21.15.63:0/3663095214 conn(0x7f3d200788f0 msgr2=0x7f3d20078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 -- 172.21.15.63:0/3663095214 shutdown_connections 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.639+0000 7f3d25800700 1 -- 172.21.15.63:0/3663095214 wait complete. 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.640+0000 7f3d25800700 1 Processor -- start 2024-04-25T18:56:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.640+0000 7f3d25800700 1 -- start start 2024-04-25T18:56:27.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:27.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d20101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3d20112c10 con 0x7f3d2006a750 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d20112a90 con 0x7f3d20108a30 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d25800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3d20112d90 con 0x7f3d2006b0f0 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d1ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d1e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d20101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d1ea00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56856/0 (socket says 172.21.15.63:56856) 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.641+0000 7f3d1ea00700 1 -- 172.21.15.63:0/4068071799 learned_addr learned my addr 172.21.15.63:0/4068071799 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1f400700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 -- 172.21.15.63:0/4068071799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 msgr2=0x7f3d20102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 -- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 msgr2=0x7f3d20101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:27.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 -- 172.21.15.63:0/4068071799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d201a7730 con 0x7f3d2006b0f0 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1f400700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1ea00700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d1e000700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d20101f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00eb90 tx=0x7f3d0c00eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.642+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c00ccf0 con 0x7f3d2006b0f0 2024-04-25T18:56:27.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.643+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d0c0044d0 con 0x7f3d2006b0f0 2024-04-25T18:56:27.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.643+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d201a7a20 con 0x7f3d2006b0f0 2024-04-25T18:56:27.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.643+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c010430 con 0x7f3d2006b0f0 2024-04-25T18:56:27.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.643+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d201a7f60 con 0x7f3d2006b0f0 2024-04-25T18:56:27.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.644+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d20069730 con 0x7f3d2006b0f0 2024-04-25T18:56:27.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.646+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3d0c003680 con 0x7f3d2006b0f0 2024-04-25T18:56:27.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.647+0000 7f3d17400700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 0x7f3d040a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:27.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.647+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7f3d0c0d0b70 con 0x7f3d2006b0f0 2024-04-25T18:56:27.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.648+0000 7f3d1ea00700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 0x7f3d040a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:27.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.648+0000 7f3d1ea00700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 0x7f3d040a6270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009680 tx=0x7f3d10019040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:27.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.650+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d0c01f340 con 0x7f3d2006b0f0 2024-04-25T18:56:27.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.935+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d2002df80 con 0x7f3d2006b0f0 2024-04-25T18:56:27.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.937+0000 7f3d17400700 1 -- 172.21.15.63:0/4068071799 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+11872 (secure 0 0 0) 0x7f3d0c09d170 con 0x7f3d2006b0f0 2024-04-25T18:56:27.939 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:56:27.939 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:56:01.166752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25669":{"gid":25669,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/1965078311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":1965078311},{"type":"v1","addr":"172.21.15.114:6835","nonce":1965078311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:56:27.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 msgr2=0x7f3d040a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:27.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 0x7f3d040a6270 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009680 tx=0x7f3d10019040 comp rx=0 tx=0).stop 2024-04-25T18:56:27.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 msgr2=0x7f3d20101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:27.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d20101f50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00eb90 tx=0x7f3d0c00eea0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 shutdown_connections 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.940+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3d040a3db0 0x7f3d040a6270 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d20108a30 0x7f3d20102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3d2006a750 0x7f3d20101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 --2- 172.21.15.63:0/4068071799 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3d2006b0f0 0x7f3d20101f50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 >> 172.21.15.63:0/4068071799 conn(0x7f3d200788f0 msgr2=0x7f3d200ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 shutdown_connections 2024-04-25T18:56:27.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:27.941+0000 7f3d25800700 1 -- 172.21.15.63:0/4068071799 wait complete. 2024-04-25T18:56:27.945 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 39 2024-04-25T18:56:28.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:28 smithi063 ceph-mon[28006]: pgmap v309: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:28.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:28 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/4068071799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:56:28.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.42 MB/sec execute 387 sec latency 43098.140 ms 2024-04-25T18:56:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:28 smithi114 ceph-mon[32988]: pgmap v309: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:28 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/4068071799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:56:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:28 smithi070 ceph-mon[32820]: pgmap v309: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 4.3 KiB/s wr, 7 op/s 2024-04-25T18:56:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:28 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/4068071799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:56:28.831 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id c393d2af --release --await' 2024-04-25T18:56:28.831 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id c393d2af --release --await 2024-04-25T18:56:29.285 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:56:29.500 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.41 MB/sec execute 388 sec latency 44098.285 ms 2024-04-25T18:56:30.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:30 smithi063 ceph-mon[28006]: pgmap v310: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T18:56:30.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.41 MB/sec execute 389 sec latency 45098.449 ms 2024-04-25T18:56:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:30 smithi114 ceph-mon[32988]: pgmap v310: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T18:56:30.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:30 smithi070 ceph-mon[32820]: pgmap v310: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 22 KiB/s wr, 13 op/s 2024-04-25T18:56:31.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311069 3.40 MB/sec execute 390 sec latency 46098.576 ms 2024-04-25T18:56:31.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 -- 172.21.15.63:0/3688907841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8104ee0 msgr2=0x7f6be81052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 --2- 172.21.15.63:0/3688907841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8104ee0 0x7f6be81052c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80098f0 tx=0x7f6bd8009c00 comp rx=0 tx=0).stop 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 -- 172.21.15.63:0/3688907841 shutdown_connections 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 --2- 172.21.15.63:0/3688907841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8109fc0 0x7f6be8111b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 --2- 172.21.15.63:0/3688907841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8104ee0 0x7f6be81052c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 --2- 172.21.15.63:0/3688907841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8105890 0x7f6be81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 -- 172.21.15.63:0/3688907841 >> 172.21.15.63:0/3688907841 conn(0x7f6be8100770 msgr2=0x7f6be8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 -- 172.21.15.63:0/3688907841 shutdown_connections 2024-04-25T18:56:31.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.885+0000 7f6bee200700 1 -- 172.21.15.63:0/3688907841 wait complete. 2024-04-25T18:56:31.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 Processor -- start 2024-04-25T18:56:31.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 -- start start 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8105890 0x7f6be819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8109fc0 0x7f6be819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6be8114e50 con 0x7f6be8104ee0 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6be8114cd0 con 0x7f6be8109fc0 2024-04-25T18:56:31.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6bee200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6be8114fd0 con 0x7f6be8105890 2024-04-25T18:56:31.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6be7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6be6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8105890 0x7f6be819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6be7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56876/0 (socket says 172.21.15.63:56876) 2024-04-25T18:56:31.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6be7400700 1 -- 172.21.15.63:0/3081490565 learned_addr learned my addr 172.21.15.63:0/3081490565 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.887+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8109fc0 0x7f6be819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 -- 172.21.15.63:0/3081490565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8105890 msgr2=0x7f6be819b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8105890 0x7f6be819b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 -- 172.21.15.63:0/3081490565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8109fc0 msgr2=0x7f6be819fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8109fc0 0x7f6be819fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 -- 172.21.15.63:0/3081490565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be81a0390 con 0x7f6be8104ee0 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6be7400700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009230 tx=0x7f6bd800f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6bdfe00700 1 -- 172.21.15.63:0/3081490565 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8027070 con 0x7f6be8104ee0 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6bdfe00700 1 -- 172.21.15.63:0/3081490565 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd8005570 con 0x7f6be8104ee0 2024-04-25T18:56:31.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be81a0620 con 0x7f6be8104ee0 2024-04-25T18:56:31.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6bdfe00700 1 -- 172.21.15.63:0/3081490565 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8004930 con 0x7f6be8104ee0 2024-04-25T18:56:31.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.888+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be81a7f00 con 0x7f6be8104ee0 2024-04-25T18:56:31.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.892+0000 7f6bdfe00700 1 -- 172.21.15.63:0/3081490565 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6bd802a070 con 0x7f6be8104ee0 2024-04-25T18:56:31.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.893+0000 7f6bf8aeab80 1 Processor -- start 2024-04-25T18:56:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.893+0000 7f6bf8aeab80 1 -- start start 2024-04-25T18:56:31.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.893+0000 7f6bdfe00700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 0x7f6bcc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bdfe00700 1 -- 172.21.15.63:0/3081490565 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7f6bd80d11d0 con 0x7f6be8104ee0 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa133c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa15cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf952f90 con 0x55f5cfa13900 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf8f9830 con 0x55f5cfa08a50 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf94ef50 con 0x55f5cfa08130 2024-04-25T18:56:31.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6be7e00700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 0x7f6bcc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6be7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6be7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45554/0 (socket says 172.21.15.63:45554) 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6be7400700 1 -- 172.21.15.63:0/1979037568 learned_addr learned my addr 172.21.15.63:0/1979037568 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.894+0000 7f6be7e00700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa15cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be6a00700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 -- 172.21.15.63:0/1979037568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 msgr2=0x55f5cfa133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 -- 172.21.15.63:0/1979037568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 msgr2=0x55f5cfa15cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa15cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 -- 172.21.15.63:0/1979037568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5cf8ef780 con 0x55f5cfa08130 2024-04-25T18:56:31.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be6a00700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7e00700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 0x7f6bcc0a6270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6be000f870 tx=0x7f6be000fc30 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6be7400700 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80996e0 tx=0x7f6bd800ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=774626465fdee678 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.895+0000 7f6bdcc00700 1 -- 172.21.15.63:0/1979037568 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80a5e60 con 0x55f5cfa08130 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.896+0000 7f6bdcc00700 1 -- 172.21.15.63:0/1979037568 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd8009480 con 0x55f5cfa08130 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.896+0000 7f6bdcc00700 1 -- 172.21.15.63:0/1979037568 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd809d4e0 con 0x55f5cfa08130 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.896+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1979037568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 msgr2=0x55f5cfa08510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.896+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80996e0 tx=0x7f6bd800ba20 comp rx=0 tx=0).stop 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1979037568 shutdown_connections 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa08510 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa15cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1979037568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1979037568 >> 172.21.15.63:0/1979037568 conn(0x55f5cf92e6a0 msgr2=0x55f5cf9fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1979037568 shutdown_connections 2024-04-25T18:56:31.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1979037568 wait complete. 2024-04-25T18:56:31.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 -- start start 2024-04-25T18:56:31.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.897+0000 7f6bf8aeab80 1 Processor -- start 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa10340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa10880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f5cfa16230 con 0x55f5cfa13900 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f5cf90c960 con 0x55f5cfa08a50 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5cfa163b0 con 0x55f5cfa08130 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bf8aeab80 10 client.0 _ll_register_callbacks cb 0x55f5cf8eb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be6a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa10340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa10880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45564/0 (socket says 172.21.15.63:45564) 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 -- 172.21.15.63:0/4065307753 learned_addr learned my addr 172.21.15.63:0/4065307753 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 -- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 msgr2=0x55f5cfa10340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa10340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 -- 172.21.15.63:0/4065307753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 msgr2=0x55f5cfa10880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa10880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 -- 172.21.15.63:0/4065307753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5cf8a09d0 con 0x55f5cfa08130 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7e00700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa10880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6be7400700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6bd809d210 tx=0x7f6bd80a1730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e73bce8871fd4c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.899+0000 7f6bd1600700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd800b680 con 0x55f5cfa08130 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7f6bd809d4e0 con 0x55f5cfa08130 2024-04-25T18:56:31.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6bd8009480 con 0x55f5cfa08130 2024-04-25T18:56:31.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6bd800b820 con 0x55f5cfa08130 2024-04-25T18:56:31.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bf8aeab80 10 client.26416 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:31.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f5cf8e9bf0 con 0x55f5cfa08130 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.900+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd8009480 con 0x55f5cfa08130 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f6bd809c6d0 con 0x55f5cfa08130 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 10 client.26416 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 10 client.26416 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 20 client.26416 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa22cd0 0x55f5cfa250c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 4 client.26416 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5cf960780 con 0x55f5cfa22cd0 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6be6a00700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa22cd0 0x55f5cfa250c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6be6a00700 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa22cd0 0x55f5cfa250c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb3c35f28f06108e server_cookie=f5e6d4bf339b4c3a in_seq=0 out_seq=0 2024-04-25T18:56:31.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.901+0000 7f6bd1600700 10 client.26416 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:31.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bd1600700 1 -- 172.21.15.63:0/4065307753 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5cf960780 con 0x55f5cfa22cd0 2024-04-25T18:56:31.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bd1600700 10 client.26416 handle_command_reply: tid=0 2024-04-25T18:56:31.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 client.26416 shutdown 2024-04-25T18:56:31.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 msgr2=0x55f5cfa12fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6bd809d210 tx=0x7f6bd80a1730 comp rx=0 tx=0).stop 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 shutdown_connections 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa22cd0 0x55f5cfa250c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08130 0x55f5cfa12fa0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa10880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/4065307753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08a50 0x55f5cfa10340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bd1600700 0 client.26416 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 >> 172.21.15.63:0/4065307753 conn(0x55f5cf92e6a0 msgr2=0x55f5cf9fb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 shutdown_connections 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/4065307753 wait complete. 2024-04-25T18:56:31.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.904+0000 7f6bf8aeab80 20 client.26416 trim_cache size 0 max 16384 2024-04-25T18:56:31.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.925+0000 7f6bf8aeab80 1 Processor -- start 2024-04-25T18:56:31.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.925+0000 7f6bf8aeab80 1 -- start start 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 0x55f5cfa0f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6be81ada70 con 0x55f5cfa08130 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6be81ab270 con 0x55f5cfa13900 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6be807b080 con 0x55f5cfa08a50 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 0x55f5cfa0f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 0x55f5cfa0f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56900/0 (socket says 172.21.15.63:56900) 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45578/0 (socket says 172.21.15.63:45578) 2024-04-25T18:56:31.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40994/0 (socket says 172.21.15.63:40994) 2024-04-25T18:56:31.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7400700 1 -- 172.21.15.63:0/2562918075 learned_addr learned my addr 172.21.15.63:0/2562918075 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 -- 172.21.15.63:0/2562918075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 msgr2=0x55f5cfa1f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 -- 172.21.15.63:0/2562918075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 msgr2=0x55f5cfa0f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 0x55f5cfa0f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 -- 172.21.15.63:0/2562918075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5cfa1f630 con 0x55f5cfa13900 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.927+0000 7f6be7e00700 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6be0018640 tx=0x7f6be001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1a594d9f50c0cf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bd3e00700 1 -- 172.21.15.63:0/2562918075 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0004db0 con 0x55f5cfa13900 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bd3e00700 1 -- 172.21.15.63:0/2562918075 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6be00089c0 con 0x55f5cfa13900 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bd3e00700 1 -- 172.21.15.63:0/2562918075 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be001a980 con 0x55f5cfa13900 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/2562918075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 msgr2=0x55f5cfa2a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6be0018640 tx=0x7f6be001f000 comp rx=0 tx=0).stop 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/2562918075 shutdown_connections 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa13900 0x55f5cfa2a8d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa08130 0x55f5cfa0f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/2562918075 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/2562918075 >> 172.21.15.63:0/2562918075 conn(0x55f5cf92e6a0 msgr2=0x55f5cfa11940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/2562918075 shutdown_connections 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.928+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/2562918075 wait complete. 2024-04-25T18:56:31.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.929+0000 7f6bf8aeab80 1 -- start start 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.929+0000 7f6bf8aeab80 1 Processor -- start 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 0x55f5cfa1b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa18670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6be81a9730 con 0x55f5cfa13900 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6be81aaba0 con 0x55f5cfa08130 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6be81a9e00 con 0x55f5cfa08a50 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6bf8aeab80 10 client.0 _ll_register_callbacks cb 0x55f5cf8eb820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa18670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be7400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 0x55f5cfa1b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be7e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa18670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56910/0 (socket says 172.21.15.63:56910) 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be7e00700 1 -- 172.21.15.63:0/1120718729 learned_addr learned my addr 172.21.15.63:0/1120718729 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be6a00700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.930+0000 7f6be7400700 1 -- 172.21.15.63:0/1120718729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 msgr2=0x55f5cfa1bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6be7400700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6be7400700 1 -- 172.21.15.63:0/1120718729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 msgr2=0x55f5cfa18670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6be7400700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa18670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6be7400700 1 -- 172.21.15.63:0/1120718729 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5cfa18eb0 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6be7400700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 0x55f5cfa1b600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8099c50 tx=0x7f6bd800ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a170a0d8ccdffade server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bd1600700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80b5040 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6be00089c0 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7f6bd8005180 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f6bd800b680 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bf8aeab80 10 client.26399 fetch_fsmap learned FSMap version 39 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.931+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55f5cfa18eb0 con 0x55f5cfa08130 2024-04-25T18:56:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bd80a9080 con 0x55f5cfa08130 2024-04-25T18:56:31.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f6bd80a5a60 con 0x55f5cfa08130 2024-04-25T18:56:31.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 10 client.26399 fetch_fsmap finished waiting for FSMap version 39 2024-04-25T18:56:31.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 10 client.26399 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:56:31.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 20 client.26399 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa89fb0 0x55f5cfa8c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 4 client.26399 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "c393d2af", "release": true, "await": true}] 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.932+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "c393d2af", "release": true, "await": true}) -- 0x55f5cf95a820 con 0x55f5cfa89fb0 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.933+0000 7f6be6a00700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa89fb0 0x55f5cfa8c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.933+0000 7f6be6a00700 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa89fb0 0x55f5cfa8c3a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc80ec37f628c554 server_cookie=7a3fb2a1eed44e78 in_seq=0 out_seq=0 2024-04-25T18:56:31.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.933+0000 7f6bd1600700 10 client.26399 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.936+0000 7f6bd1600700 1 -- 172.21.15.63:0/1120718729 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55f5cf95a820 con 0x55f5cfa89fb0 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.936+0000 7f6bd1600700 10 client.26399 handle_command_reply: tid=0 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.936+0000 7f6bf8aeab80 1 client.26399 shutdown 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 msgr2=0x55f5cfa1b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 0x55f5cfa1b600 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8099c50 tx=0x7f6bd800ba20 comp rx=0 tx=0).stop 2024-04-25T18:56:31.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 shutdown_connections 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5cfa89fb0 0x55f5cfa8c3a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5cfa13900 0x55f5cfa18670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5cfa08130 0x55f5cfa1b600 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 --2- 172.21.15.63:0/1120718729 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5cfa08a50 0x55f5cfa1bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 >> 172.21.15.63:0/1120718729 conn(0x55f5cf92e6a0 msgr2=0x55f5cfa0a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bd1600700 0 client.26399 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 shutdown_connections 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 1 -- 172.21.15.63:0/1120718729 wait complete. 2024-04-25T18:56:31.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.937+0000 7f6bf8aeab80 20 client.26399 trim_cache size 0 max 16384 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 39, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 32, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "c393d2af": { 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 32, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:56:31.941 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 84.8, 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:56:31.942 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 msgr2=0x7f6bcc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 0x7f6bcc0a6270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6be000f870 tx=0x7f6be000fc30 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 msgr2=0x7f6be819b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8009230 tx=0x7f6bd800f9b0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 shutdown_connections 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6bcc0a3db0 0x7f6bcc0a6270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6be8109fc0 0x7f6be819fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6be8104ee0 0x7f6be819b380 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 --2- 172.21.15.63:0/3081490565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6be8105890 0x7f6be819b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.938+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 >> 172.21.15.63:0/3081490565 conn(0x7f6be8100770 msgr2=0x7f6be8109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:31.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.939+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 shutdown_connections 2024-04-25T18:56:31.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:56:31.939+0000 7f6bee200700 1 -- 172.21.15.63:0/3081490565 wait complete. 2024-04-25T18:56:32.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311104 3.39 MB/sec execute 391 sec latency 46542.837 ms 2024-04-25T18:56:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:32 smithi114 ceph-mon[32988]: pgmap v311: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:32 smithi063 ceph-mon[28006]: pgmap v311: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:32 smithi070 ceph-mon[32820]: pgmap v311: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:32.834 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'c393d2af', seconds elapsed: 55.5 2024-04-25T18:56:33.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311104 3.38 MB/sec execute 392 sec latency 1526.911 ms 2024-04-25T18:56:34.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311104 3.37 MB/sec execute 393 sec latency 2527.013 ms 2024-04-25T18:56:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:34 smithi114 ceph-mon[32988]: pgmap v312: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:34 smithi063 ceph-mon[28006]: pgmap v312: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:34.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:34 smithi070 ceph-mon[32820]: pgmap v312: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 8 op/s 2024-04-25T18:56:35.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311104 3.36 MB/sec execute 394 sec latency 3527.148 ms 2024-04-25T18:56:36.501 INFO:tasks.workunit.client.0.smithi063.stdout: 1 311528 3.37 MB/sec execute 395 sec latency 4212.528 ms 2024-04-25T18:56:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:36 smithi114 ceph-mon[32988]: pgmap v313: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-25T18:56:36.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:36 smithi063 ceph-mon[28006]: pgmap v313: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-25T18:56:36.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:36 smithi070 ceph-mon[32820]: pgmap v313: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-25T18:56:37.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 312459 3.37 MB/sec execute 396 sec latency 52.974 ms 2024-04-25T18:56:37.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:37 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:37.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:37 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:37.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:37 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:38.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 313134 3.37 MB/sec execute 397 sec latency 19.467 ms 2024-04-25T18:56:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:38 smithi114 ceph-mon[32988]: pgmap v314: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-25T18:56:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:38 smithi063 ceph-mon[28006]: pgmap v314: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-25T18:56:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 -- 172.21.15.63:0/3547703356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 msgr2=0x7fca50111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 --2- 172.21.15.63:0/3547703356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca50111040 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fca48009900 tx=0x7fca4802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 -- 172.21.15.63:0/3547703356 shutdown_connections 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 --2- 172.21.15.63:0/3547703356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca50104d50 0x7fca50108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 --2- 172.21.15.63:0/3547703356 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca50111040 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 --2- 172.21.15.63:0/3547703356 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca501043a0 0x7fca50104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 -- 172.21.15.63:0/3547703356 >> 172.21.15.63:0/3547703356 conn(0x7fca500ffc40 msgr2=0x7fca50102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 -- 172.21.15.63:0/3547703356 shutdown_connections 2024-04-25T18:56:38.639 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.637+0000 7fca54a00640 1 -- 172.21.15.63:0/3547703356 wait complete. 2024-04-25T18:56:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.638+0000 7fca54a00640 1 Processor -- start 2024-04-25T18:56:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.638+0000 7fca54a00640 1 -- start start 2024-04-25T18:56:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca501043a0 0x7fca501a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca5019a670 0x7fca5019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca50114410 con 0x7fca5019a670 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca50114290 con 0x7fca501043a0 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca54a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca50114590 con 0x7fca50109470 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58122/0 (socket says 172.21.15.63:58122) 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 -- 172.21.15.63:0/730916211 learned_addr learned my addr 172.21.15.63:0/730916211 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4de00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca501043a0 0x7fca501a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4e800640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca5019a670 0x7fca5019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 -- 172.21.15.63:0/730916211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca501043a0 msgr2=0x7fca501a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca501043a0 0x7fca501a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 -- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca5019a670 msgr2=0x7fca5019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca5019a670 0x7fca5019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.639+0000 7fca4d400640 1 -- 172.21.15.63:0/730916211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5019b270 con 0x7fca50109470 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca4d400640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca46a00640 1 -- 172.21.15.63:0/730916211 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3800cd80 con 0x7fca50109470 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca46a00640 1 -- 172.21.15.63:0/730916211 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca380054d0 con 0x7fca50109470 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca46a00640 1 -- 172.21.15.63:0/730916211 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38010430 con 0x7fca50109470 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca501a7350 con 0x7fca50109470 2024-04-25T18:56:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.640+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca501a7700 con 0x7fca50109470 2024-04-25T18:56:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.645+0000 7fca57817740 1 Processor -- start 2024-04-25T18:56:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.645+0000 7fca46a00640 1 -- 172.21.15.63:0/730916211 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fca38004fd0 con 0x7fca50109470 2024-04-25T18:56:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 -- start start 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05b140 0x5625df0654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df067df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5625defc0900 con 0x5625df05b140 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625def66ef0 con 0x5625df065a00 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca57817740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5625defbc8c0 con 0x5625df05a820 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca4e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca4de00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05b140 0x5625df0654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca4e800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58132/0 (socket says 172.21.15.63:58132) 2024-04-25T18:56:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca4e800640 1 -- 172.21.15.63:0/3592643136 learned_addr learned my addr 172.21.15.63:0/3592643136 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.646+0000 7fca4d400640 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df067df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 -- 172.21.15.63:0/3592643136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 msgr2=0x5625df067df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df067df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 -- 172.21.15.63:0/3592643136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05b140 msgr2=0x5625df0654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05b140 0x5625df0654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 -- 172.21.15.63:0/3592643136 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625decd3210 con 0x5625df05a820 2024-04-25T18:56:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca4e800640 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca4802a9e0 tx=0x7fca48005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b405c679fe3e931d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca3f000640 1 -- 172.21.15.63:0/3592643136 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48040070 con 0x5625df05a820 2024-04-25T18:56:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca3f000640 1 -- 172.21.15.63:0/3592643136 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca480044c0 con 0x5625df05a820 2024-04-25T18:56:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.647+0000 7fca3f000640 1 -- 172.21.15.63:0/3592643136 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca48035040 con 0x5625df05a820 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca46a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 0x7fca340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca46a00640 1 -- 172.21.15.63:0/730916211 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fca380d0b60 con 0x7fca50109470 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca4de00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 0x7fca340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 -- 172.21.15.63:0/3592643136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 msgr2=0x5625df05ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca4802a9e0 tx=0x7fca48005be0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 -- 172.21.15.63:0/3592643136 shutdown_connections 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df067df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05b140 0x5625df0654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 --2- 172.21.15.63:0/3592643136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05a820 0x5625df05ac00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.648+0000 7fca57817740 1 -- 172.21.15.63:0/3592643136 >> 172.21.15.63:0/3592643136 conn(0x5625df056d30 msgr2=0x5625df057110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.649+0000 7fca57817740 1 -- 172.21.15.63:0/3592643136 shutdown_connections 2024-04-25T18:56:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.649+0000 7fca57817740 1 -- 172.21.15.63:0/3592643136 wait complete. 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.649+0000 7fca4de00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 0x7fca340a6300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5625df05bdc0 tx=0x7fca40009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.649+0000 7fca57817740 1 -- start start 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.649+0000 7fca57817740 1 Processor -- start 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df062a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df0634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5625df0685d0 con 0x5625df05a820 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5625def7a310 con 0x5625df065a00 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5625df068750 con 0x5625df05b140 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca57817740 10 client.0 _ll_register_callbacks cb 0x5625def48cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df0634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58140/0 (socket says 172.21.15.63:58140) 2024-04-25T18:56:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4d400640 1 -- 172.21.15.63:0/3716392407 learned_addr learned my addr 172.21.15.63:0/3716392407 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4de00640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df062a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4d400640 1 -- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 msgr2=0x5625df0634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.650+0000 7fca4d400640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df0634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca4d400640 1 -- 172.21.15.63:0/3716392407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 msgr2=0x5625df062a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca4d400640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df062a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca4d400640 1 -- 172.21.15.63:0/3716392407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625ded9f620 con 0x5625df05b140 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca4e800640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df0634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca4d400640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fca3809c9f0 tx=0x7fca380d4200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3efc3301430146d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca33400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3809c5a0 con 0x5625df05b140 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fca3809ad60 con 0x5625df05b140 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5625defd48b0 con 0x5625df05b140 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fca3809c3c0 con 0x5625df05b140 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.651+0000 7fca57817740 10 client.26434 fetch_fsmap learned FSMap version 40 2024-04-25T18:56:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5625df0748c0 con 0x5625df05b140 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5625defd48b0 con 0x5625df05b140 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fca380ce490 con 0x5625df05b140 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 10 client.26434 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 10 client.26434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 20 client.26434 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df078220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 4 client.26434 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625defce320 con 0x5625df075dd0 2024-04-25T18:56:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.652+0000 7fca4de00640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df078220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.653+0000 7fca4de00640 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df078220 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43376a4749685911 server_cookie=ebfba58b7c780105 in_seq=0 out_seq=0 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.653+0000 7fca33400640 10 client.26434 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca33400640 1 -- 172.21.15.63:0/3716392407 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5625defce320 con 0x5625df075dd0 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca33400640 10 client.26434 handle_command_reply: tid=0 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 client.26434 shutdown 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 msgr2=0x5625df062f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fca3809c9f0 tx=0x7fca380d4200 comp rx=0 tx=0).stop 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 shutdown_connections 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df078220 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca33400640 0 client.26434 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df0634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df062f70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 --2- 172.21.15.63:0/3716392407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df062a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 >> 172.21.15.63:0/3716392407 conn(0x5625df056d30 msgr2=0x5625df05cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 shutdown_connections 2024-04-25T18:56:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 1 -- 172.21.15.63:0/3716392407 wait complete. 2024-04-25T18:56:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.654+0000 7fca57817740 20 client.26434 trim_cache size 0 max 16384 2024-04-25T18:56:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.672+0000 7fca57817740 1 Processor -- start 2024-04-25T18:56:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.672+0000 7fca57817740 1 -- start start 2024-04-25T18:56:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df071490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df07e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca501b26d0 con 0x5625df05a820 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca501afee0 con 0x5625df065a00 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca57817740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca501add50 con 0x5625df05b140 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4e800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df07e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58152/0 (socket says 172.21.15.63:58152) 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 -- 172.21.15.63:0/3780280389 learned_addr learned my addr 172.21.15.63:0/3780280389 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4de00640 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 -- 172.21.15.63:0/3780280389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 msgr2=0x5625df07e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df07e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 -- 172.21.15.63:0/3780280389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 msgr2=0x5625df071490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.673+0000 7fca4d400640 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df071490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.674+0000 7fca4d400640 1 -- 172.21.15.63:0/3780280389 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625df05bad0 con 0x5625df05b140 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.674+0000 7fca4d400640 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fca3809ad10 tx=0x7fca38005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5185933aece0f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.674+0000 7fca3e600640 1 -- 172.21.15.63:0/3780280389 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3809c5a0 con 0x5625df05b140 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.674+0000 7fca3e600640 1 -- 172.21.15.63:0/3780280389 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca380046c0 con 0x5625df05b140 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.674+0000 7fca3e600640 1 -- 172.21.15.63:0/3780280389 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3809ca60 con 0x5625df05b140 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- 172.21.15.63:0/3780280389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 msgr2=0x5625df07db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fca3809ad10 tx=0x7fca38005e30 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- 172.21.15.63:0/3780280389 shutdown_connections 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df065a00 0x5625df07e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df05b140 0x5625df07db30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 --2- 172.21.15.63:0/3780280389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- 172.21.15.63:0/3780280389 >> 172.21.15.63:0/3780280389 conn(0x5625df056d30 msgr2=0x5625df057110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- 172.21.15.63:0/3780280389 shutdown_connections 2024-04-25T18:56:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- 172.21.15.63:0/3780280389 wait complete. 2024-04-25T18:56:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 -- start start 2024-04-25T18:56:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.675+0000 7fca57817740 1 Processor -- start 2024-04-25T18:56:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df06da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 0x5625df06b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca501ae3a0 con 0x5625df05a820 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca501af810 con 0x5625df05b140 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fca501aea70 con 0x5625df065a00 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca57817740 10 client.0 _ll_register_callbacks cb 0x7fca380048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca4d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca4d400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33264/0 (socket says 172.21.15.63:33264) 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca4d400640 1 -- 172.21.15.63:0/490110289 learned_addr learned my addr 172.21.15.63:0/490110289 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.676+0000 7fca4de00640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df06da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4e800640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 0x5625df06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 -- 172.21.15.63:0/490110289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 msgr2=0x5625df06b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 0x5625df06b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 -- 172.21.15.63:0/490110289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 msgr2=0x5625df06da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df06da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 -- 172.21.15.63:0/490110289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625df06b990 con 0x5625df05b140 2024-04-25T18:56:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4e800640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 0x5625df06b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca4d400640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca38098ec0 tx=0x7fca3809d280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48c52cff6f071199 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca33400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3809c5a0 con 0x5625df05b140 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 8587+0+0 (secure 0 0 0) 0x7fca3809c0b0 con 0x5625df05b140 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5625defc8190 con 0x5625df05b140 2024-04-25T18:56:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fca380a0e10 con 0x5625df05b140 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca57817740 10 client.26417 fetch_fsmap learned FSMap version 40 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.677+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5625df0dc0f0 con 0x5625df05b140 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5625defc8190 con 0x5625df05b140 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 40) ==== 8354+0+0 (secure 0 0 0) 0x7fca380872b0 con 0x5625df05b140 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 10 client.26417 fetch_fsmap finished waiting for FSMap version 40 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 10 client.26417 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 20 client.26417 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df0de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 4 client.26417 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625def444b0 con 0x5625df075dd0 2024-04-25T18:56:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.678+0000 7fca4de00640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df0de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.679+0000 7fca4de00640 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df0de7a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3aebbe8fadb07b server_cookie=88320878f8d080d in_seq=0 out_seq=0 2024-04-25T18:56:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.679+0000 7fca33400640 10 client.26417 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.695+0000 7fca33400640 1 -- 172.21.15.63:0/490110289 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625def444b0 con 0x5625df075dd0 2024-04-25T18:56:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.695+0000 7fca33400640 10 client.26417 handle_command_reply: tid=0 2024-04-25T18:56:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.695+0000 7fca57817740 1 client.26417 shutdown 2024-04-25T18:56:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 msgr2=0x5625df06af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca38098ec0 tx=0x7fca3809d280 comp rx=0 tx=0).stop 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 shutdown_connections 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5625df075dd0 0x5625df0de7a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5625df05b140 0x5625df06af10 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5625df065a00 0x5625df06b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 --2- 172.21.15.63:0/490110289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625df05a820 0x5625df06da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca33400640 0 client.26417 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 >> 172.21.15.63:0/490110289 conn(0x5625df056d30 msgr2=0x5625df05c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 shutdown_connections 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 1 -- 172.21.15.63:0/490110289 wait complete. 2024-04-25T18:56:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.696+0000 7fca57817740 20 client.26417 trim_cache size 0 max 16384 2024-04-25T18:56:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:56:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:56:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:56:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 msgr2=0x7fca340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 0x7fca340a6300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x5625df05bdc0 tx=0x7fca40009500 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 msgr2=0x7fca501a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fca3800e9c0 tx=0x7fca3800ee80 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 shutdown_connections 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fca340a3e40 0x7fca340a6300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca5019a670 0x7fca5019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fca50109470 0x7fca501a0af0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 --2- 172.21.15.63:0/730916211 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca501043a0 0x7fca501a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 >> 172.21.15.63:0/730916211 conn(0x7fca500ffc40 msgr2=0x7fca501006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 shutdown_connections 2024-04-25T18:56:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:38.697+0000 7fca54a00640 1 -- 172.21.15.63:0/730916211 wait complete. 2024-04-25T18:56:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:38 smithi070 ceph-mon[32820]: pgmap v314: 97 pgs: 97 active+clean; 90 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 29 KiB/s wr, 9 op/s 2024-04-25T18:56:39.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:39 smithi063 ceph-mon[28006]: daemon mds.f restarted 2024-04-25T18:56:39.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 314070 3.37 MB/sec execute 398 sec latency 26.936 ms 2024-04-25T18:56:39.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:39 smithi114 ceph-mon[32988]: daemon mds.f restarted 2024-04-25T18:56:39.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:56:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setuser ceph since I am not root 2024-04-25T18:56:39.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:56:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setgroup ceph since I am not root 2024-04-25T18:56:39.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:56:39 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/2314696574 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559c6ef19a40 con 0x559c6eeec800 2024-04-25T18:56:39.568 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 18:56:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T18:56:39.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:39 smithi070 ceph-mon[32820]: daemon mds.f restarted 2024-04-25T18:56:40.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 315518 3.39 MB/sec execute 399 sec latency 20.037 ms 2024-04-25T18:56:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: pgmap v315: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 566 KiB/s wr, 54 op/s 2024-04-25T18:56:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T18:56:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/2546666842,v1:172.21.15.114:6835/2546666842] up:boot 2024-04-25T18:56:40.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:56:40.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:56:40.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:40 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: pgmap v315: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 566 KiB/s wr, 54 op/s 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/2546666842,v1:172.21.15.114:6835/2546666842] up:boot 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:56:40.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:40 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: pgmap v315: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 566 KiB/s wr, 54 op/s 2024-04-25T18:56:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: osdmap e107: 12 total, 12 up, 12 in 2024-04-25T18:56:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/2546666842,v1:172.21.15.114:6835/2546666842] up:boot 2024-04-25T18:56:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T18:56:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T18:56:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:40 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:56:41.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 316262 3.39 MB/sec execute 400 sec latency 13.582 ms 2024-04-25T18:56:42.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 317200 3.39 MB/sec execute 401 sec latency 24.291 ms 2024-04-25T18:56:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:42 smithi114 ceph-mon[32988]: pgmap v317: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:42 smithi063 ceph-mon[28006]: pgmap v317: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:42 smithi070 ceph-mon[32820]: pgmap v317: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:43.502 INFO:tasks.workunit.client.0.smithi063.stdout: 1 318312 3.39 MB/sec execute 402 sec latency 22.804 ms 2024-04-25T18:56:44.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 319530 3.41 MB/sec execute 403 sec latency 23.917 ms 2024-04-25T18:56:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:44 smithi114 ceph-mon[32988]: pgmap v318: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:44 smithi063 ceph-mon[28006]: pgmap v318: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:44 smithi070 ceph-mon[32820]: pgmap v318: 97 pgs: 97 active+clean; 92 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 661 KiB/s wr, 59 op/s 2024-04-25T18:56:45.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 320227 3.41 MB/sec execute 404 sec latency 24.737 ms 2024-04-25T18:56:46.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 321446 3.41 MB/sec execute 405 sec latency 45.190 ms 2024-04-25T18:56:46.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:46 smithi114 ceph-mon[32988]: pgmap v319: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:46 smithi063 ceph-mon[28006]: pgmap v319: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:46 smithi070 ceph-mon[32820]: pgmap v319: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:47.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 322774 3.43 MB/sec execute 406 sec latency 15.335 ms 2024-04-25T18:56:48.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 323368 3.43 MB/sec execute 407 sec latency 16.975 ms 2024-04-25T18:56:48.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:48 smithi114 ceph-mon[32988]: pgmap v320: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:48 smithi063 ceph-mon[28006]: pgmap v320: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:48 smithi070 ceph-mon[32820]: pgmap v320: 97 pgs: 97 active+clean; 94 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 140 op/s 2024-04-25T18:56:49.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 324372 3.43 MB/sec execute 408 sec latency 43.464 ms 2024-04-25T18:56:50.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 325709 3.44 MB/sec execute 409 sec latency 23.833 ms 2024-04-25T18:56:50.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:50 smithi114 ceph-mon[32988]: pgmap v321: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 170 op/s 2024-04-25T18:56:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:50 smithi063 ceph-mon[28006]: pgmap v321: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 170 op/s 2024-04-25T18:56:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:50 smithi070 ceph-mon[32820]: pgmap v321: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.1 MiB/s wr, 170 op/s 2024-04-25T18:56:51.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 326700 3.45 MB/sec execute 410 sec latency 19.565 ms 2024-04-25T18:56:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 -- 172.21.15.63:0/417361554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074ca0 msgr2=0x7fe32410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 --2- 172.21.15.63:0/417361554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074ca0 0x7fe32410e050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe314009900 tx=0x7fe31402a480 comp rx=0 tx=0).stop 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 -- 172.21.15.63:0/417361554 shutdown_connections 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 --2- 172.21.15.63:0/417361554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe32410e590 0x7fe3241109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 --2- 172.21.15.63:0/417361554 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074ca0 0x7fe32410e050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 --2- 172.21.15.63:0/417361554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe324074380 0x7fe324074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 -- 172.21.15.63:0/417361554 >> 172.21.15.63:0/417361554 conn(0x7fe3240fd8a0 msgr2=0x7fe3240ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 -- 172.21.15.63:0/417361554 shutdown_connections 2024-04-25T18:56:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.843+0000 7fe329e00640 1 -- 172.21.15.63:0/417361554 wait complete. 2024-04-25T18:56:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.844+0000 7fe329e00640 1 Processor -- start 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.844+0000 7fe329e00640 1 -- start start 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe324074ca0 0x7fe3241a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe32410e590 0x7fe32419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe324114200 con 0x7fe32410e590 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe324114080 con 0x7fe324074ca0 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe324114380 con 0x7fe324074380 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51442/0 (socket says 172.21.15.63:51442) 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe328a00640 1 -- 172.21.15.63:0/3611083660 learned_addr learned my addr 172.21.15.63:0/3611083660 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe329400640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe32410e590 0x7fe32419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.845+0000 7fe323e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe324074ca0 0x7fe3241a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 -- 172.21.15.63:0/3611083660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe324074ca0 msgr2=0x7fe3241a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe324074ca0 0x7fe3241a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 -- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe32410e590 msgr2=0x7fe32419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe32410e590 0x7fe32419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 -- 172.21.15.63:0/3611083660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32419af50 con 0x7fe324074380 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe328a00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00db80 tx=0x7fe31c00b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe321600640 1 -- 172.21.15.63:0/3611083660 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c017070 con 0x7fe324074380 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe321600640 1 -- 172.21.15.63:0/3611083660 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe31c003b70 con 0x7fe324074380 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32419b240 con 0x7fe324074380 2024-04-25T18:56:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.846+0000 7fe321600640 1 -- 172.21.15.63:0/3611083660 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0106c0 con 0x7fe324074380 2024-04-25T18:56:51.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.847+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3241a7500 con 0x7fe324074380 2024-04-25T18:56:51.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.850+0000 7fe32e247740 1 Processor -- start 2024-04-25T18:56:51.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.850+0000 7fe32e247740 1 -- start start 2024-04-25T18:56:51.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.850+0000 7fe321600640 1 -- 172.21.15.63:0/3611083660 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe31c004020 con 0x7fe324074380 2024-04-25T18:56:51.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa64650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56384f9bf900 con 0x56384fa599b0 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56384f965ef0 con 0x56384fa64b90 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe32e247740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56384f9bb8c0 con 0x56384fa5a2d0 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe323e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa64650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe329400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe323e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59672/0 (socket says 172.21.15.63:59672) 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe329400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33232/0 (socket says 172.21.15.63:33232) 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.851+0000 7fe328a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa64650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51448/0 (socket says 172.21.15.63:51448) 2024-04-25T18:56:51.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 -- 172.21.15.63:0/2181556746 learned_addr learned my addr 172.21.15.63:0/2181556746 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 -- 172.21.15.63:0/2181556746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 msgr2=0x56384fa64650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa64650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 -- 172.21.15.63:0/2181556746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 msgr2=0x56384fa59d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe323e00640 1 -- 172.21.15.63:0/2181556746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384f6d2210 con 0x56384fa64b90 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.852+0000 7fe329400640 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe323e00640 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe314004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f39dfe8a47401d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe321600640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 0x7fe3080a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe319c00640 1 -- 172.21.15.63:0/2181556746 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314035040 con 0x56384fa64b90 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe321600640 1 -- 172.21.15.63:0/3611083660 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8543+0+0 (secure 0 0 0) 0x7fe31c0d0800 con 0x7fe324074380 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe319c00640 1 -- 172.21.15.63:0/2181556746 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3140054f0 con 0x56384fa64b90 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe319c00640 1 -- 172.21.15.63:0/2181556746 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3140035f0 con 0x56384fa64b90 2024-04-25T18:56:51.855 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.853+0000 7fe329400640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 0x7fe3080a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.857 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 -- 172.21.15.63:0/2181556746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 msgr2=0x56384fa66f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.857 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe314000c00 tx=0x7fe314004170 comp rx=0 tx=0).stop 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe329400640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 0x7fe3080a61d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009ef0 tx=0x7fe30c005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 -- 172.21.15.63:0/2181556746 shutdown_connections 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa66f80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa64650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 --2- 172.21.15.63:0/2181556746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa59d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 -- 172.21.15.63:0/2181556746 >> 172.21.15.63:0/2181556746 conn(0x56384fa55ec0 msgr2=0x56384fa562a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 -- 172.21.15.63:0/2181556746 shutdown_connections 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.854+0000 7fe32e247740 1 -- 172.21.15.63:0/2181556746 wait complete. 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.855+0000 7fe32e247740 1 -- start start 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.855+0000 7fe32e247740 1 Processor -- start 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa61b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa5a2d0 0x56384fa620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56384fa67760 con 0x56384fa599b0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56384f979310 con 0x56384fa5a2d0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56384fa678e0 con 0x56384fa64b90 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe32e247740 10 client.0 _ll_register_callbacks cb 0x56384f947cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51458/0 (socket says 172.21.15.63:51458) 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 -- 172.21.15.63:0/3514184600 learned_addr learned my addr 172.21.15.63:0/3514184600 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe323e00640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa5a2d0 0x56384fa620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe328a00640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa61b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 -- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa5a2d0 msgr2=0x56384fa620c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa5a2d0 0x56384fa620c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 -- 172.21.15.63:0/3514184600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 msgr2=0x56384fa61b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa61b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.860 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.856+0000 7fe329400640 1 -- 172.21.15.63:0/3514184600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384f79e620 con 0x56384fa64b90 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.857+0000 7fe329400640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005bc0 tx=0x7fe30c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8c7651af263dc20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.857+0000 7fe312000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.857+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c009830 con 0x56384fa64b90 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.857+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8543+0+0 (secure 0 0 0) 0x7fe30c005020 con 0x56384fa64b90 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.857+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3140054f0 con 0x56384fa64b90 2024-04-25T18:56:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe30c005220 con 0x56384fa64b90 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 10 client.26455 fetch_fsmap learned FSMap version 42 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56384fa73a80 con 0x56384fa64b90 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3140054f0 con 0x56384fa64b90 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7fe30c027070 con 0x56384fa64b90 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 10 client.26455 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 10 client.26455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 20 client.26455 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fa75000 0x56384fa77450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 4 client.26455 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.858+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56384f9d38b0 con 0x56384fa75000 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.859+0000 7fe328a00640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fa75000 0x56384fa77450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.859+0000 7fe328a00640 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fa75000 0x56384fa77450 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2397d016f99b188 server_cookie=d8a8829e5b904c0b in_seq=0 out_seq=0 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.859+0000 7fe312000640 10 client.26455 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe312000640 1 -- 172.21.15.63:0/3514184600 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56384f9d38b0 con 0x56384fa75000 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe312000640 10 client.26455 handle_command_reply: tid=0 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 client.26455 shutdown 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 msgr2=0x56384fa62600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005bc0 tx=0x7fe30c004730 comp rx=0 tx=0).stop 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 shutdown_connections 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fa75000 0x56384fa77450 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa5a2d0 0x56384fa620c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa62600 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe312000640 0 client.26455 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.860+0000 7fe32e247740 1 --2- 172.21.15.63:0/3514184600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa61b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.861+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 >> 172.21.15.63:0/3514184600 conn(0x56384fa55ec0 msgr2=0x56384fa5bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.861+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 shutdown_connections 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.861+0000 7fe32e247740 1 -- 172.21.15.63:0/3514184600 wait complete. 2024-04-25T18:56:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.861+0000 7fe32e247740 20 client.26455 trim_cache size 0 max 16384 2024-04-25T18:56:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.892+0000 7fe32e247740 1 Processor -- start 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.892+0000 7fe32e247740 1 -- start start 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa70620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa5c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa7cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241b2600 con 0x56384fa599b0 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241afe10 con 0x56384fa64b90 2024-04-25T18:56:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe32e247740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241adc60 con 0x56384fa5a2d0 2024-04-25T18:56:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe328a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa70620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe323e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa5c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa7cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe328a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa70620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33246/0 (socket says 172.21.15.63:33246) 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe328a00640 1 -- 172.21.15.63:0/2658475922 learned_addr learned my addr 172.21.15.63:0/2658475922 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 -- 172.21.15.63:0/2658475922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 msgr2=0x56384fa5c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa5c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 -- 172.21.15.63:0/2658475922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 msgr2=0x56384fa70620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa70620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.893+0000 7fe329400640 1 -- 172.21.15.63:0/2658475922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56384fa671b0 con 0x56384fa64b90 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.894+0000 7fe329400640 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa7cd60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe30c01f880 tx=0x7fe30c004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b209f0fe8d584e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.894+0000 7fe319200640 1 -- 172.21.15.63:0/2658475922 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0055c0 con 0x56384fa64b90 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.894+0000 7fe319200640 1 -- 172.21.15.63:0/2658475922 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30c003e50 con 0x56384fa64b90 2024-04-25T18:56:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.894+0000 7fe319200640 1 -- 172.21.15.63:0/2658475922 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c005010 con 0x56384fa64b90 2024-04-25T18:56:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 -- 172.21.15.63:0/2658475922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 msgr2=0x56384fa7cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa7cd60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe30c01f880 tx=0x7fe30c004610 comp rx=0 tx=0).stop 2024-04-25T18:56:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 -- 172.21.15.63:0/2658475922 shutdown_connections 2024-04-25T18:56:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa64b90 0x56384fa7cd60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa5a2d0 0x56384fa5c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 --2- 172.21.15.63:0/2658475922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa599b0 0x56384fa70620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 -- 172.21.15.63:0/2658475922 >> 172.21.15.63:0/2658475922 conn(0x56384fa55ec0 msgr2=0x56384fa562a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 -- 172.21.15.63:0/2658475922 shutdown_connections 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.895+0000 7fe32e247740 1 -- 172.21.15.63:0/2658475922 wait complete. 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 -- start start 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 Processor -- start 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa5a2d0 0x56384fa6a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa6aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241ae2d0 con 0x56384fa5a2d0 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241af740 con 0x56384fa599b0 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3241ae9a0 con 0x56384fa64b90 2024-04-25T18:56:51.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.896+0000 7fe32e247740 10 client.0 _ll_register_callbacks cb 0x7fe30c0044d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe329400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa6aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe323e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa5a2d0 0x56384fa6a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59696/0 (socket says 172.21.15.63:59696) 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 -- 172.21.15.63:0/74077021 learned_addr learned my addr 172.21.15.63:0/74077021 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 -- 172.21.15.63:0/74077021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 msgr2=0x56384fa6aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa6aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 -- 172.21.15.63:0/74077021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa5a2d0 msgr2=0x56384fa6a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa5a2d0 0x56384fa6a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 -- 172.21.15.63:0/74077021 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56384fada5e0 con 0x56384fa599b0 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe329400640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa6aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:56:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe328a00640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe31c099100 tx=0x7fe31c087040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2fd4637d85c4af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe312000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.897+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c00be60 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 8543+0+0 (secure 0 0 0) 0x7fe31c09a7d0 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe30c003e50 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe31c09ce30 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 10 client.26441 fetch_fsmap learned FSMap version 42 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x56384fadb7d0 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe30c003e50 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7fe31c023060 con 0x56384fa599b0 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 10 client.26441 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 10 client.26441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 20 client.26441 populate_metadata read hostname 'smithi063' 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fadbf00 0x56384fade320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:56:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 4 client.26441 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:56:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.898+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56384f9cd320 con 0x56384fadbf00 2024-04-25T18:56:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.899+0000 7fe323e00640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fadbf00 0x56384fade320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:56:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.899+0000 7fe323e00640 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fadbf00 0x56384fade320 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f53dddc58c855cd server_cookie=f5325e08a6b1559a in_seq=0 out_seq=0 2024-04-25T18:56:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.899+0000 7fe312000640 10 client.26441 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe312000640 1 -- 172.21.15.63:0/74077021 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56384f9cd320 con 0x56384fadbf00 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe312000640 10 client.26441 handle_command_reply: tid=0 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 client.26441 shutdown 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 msgr2=0x56384fa6a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe31c099100 tx=0x7fe31c087040 comp rx=0 tx=0).stop 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 shutdown_connections 2024-04-25T18:56:51.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56384fadbf00 0x56384fade320 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56384fa64b90 0x56384fa6aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56384fa5a2d0 0x56384fa6a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe312000640 0 client.26441 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 --2- 172.21.15.63:0/74077021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56384fa599b0 0x56384fa6a460 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.928+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 >> 172.21.15.63:0/74077021 conn(0x56384fa55ec0 msgr2=0x56384fa54790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.929+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 shutdown_connections 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.929+0000 7fe32e247740 1 -- 172.21.15.63:0/74077021 wait complete. 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.929+0000 7fe32e247740 20 client.26441 trim_cache size 0 max 16384 2024-04-25T18:56:51.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:56:51.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:56:51.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:56:51.932 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 msgr2=0x7fe3080a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 0x7fe3080a61d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009ef0 tx=0x7fe30c005f90 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 msgr2=0x7fe3241a0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00db80 tx=0x7fe31c00b630 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 shutdown_connections 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe3080a3d10 0x7fe3080a61d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe324074ca0 0x7fe3241a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe32410e590 0x7fe32419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 --2- 172.21.15.63:0/3611083660 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe324074380 0x7fe3241a0580 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 >> 172.21.15.63:0/3611083660 conn(0x7fe3240fd8a0 msgr2=0x7fe3240fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 shutdown_connections 2024-04-25T18:56:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:56:51.930+0000 7fe329e00640 1 -- 172.21.15.63:0/3611083660 wait complete. 2024-04-25T18:56:52.503 INFO:tasks.workunit.client.0.smithi063.stdout: 1 327521 3.44 MB/sec execute 411 sec latency 18.675 ms 2024-04-25T18:56:52.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:52 smithi114 ceph-mon[32988]: pgmap v322: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 954 KiB/s wr, 143 op/s 2024-04-25T18:56:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:52 smithi063 ceph-mon[28006]: pgmap v322: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 954 KiB/s wr, 143 op/s 2024-04-25T18:56:52.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:52 smithi070 ceph-mon[32820]: pgmap v322: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 954 KiB/s wr, 143 op/s 2024-04-25T18:56:53.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 328830 3.45 MB/sec execute 412 sec latency 22.983 ms 2024-04-25T18:56:54.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 330110 3.47 MB/sec execute 413 sec latency 20.891 ms 2024-04-25T18:56:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:54 smithi063 ceph-mon[28006]: pgmap v323: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 945 KiB/s wr, 142 op/s 2024-04-25T18:56:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:54 smithi070 ceph-mon[32820]: pgmap v323: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 945 KiB/s wr, 142 op/s 2024-04-25T18:56:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:54 smithi114 ceph-mon[32988]: pgmap v323: 97 pgs: 97 active+clean; 99 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 945 KiB/s wr, 142 op/s 2024-04-25T18:56:55.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 330677 3.46 MB/sec execute 414 sec latency 24.056 ms 2024-04-25T18:56:56.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 331775 3.46 MB/sec execute 415 sec latency 24.674 ms 2024-04-25T18:56:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:56 smithi063 ceph-mon[28006]: pgmap v324: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 224 op/s 2024-04-25T18:56:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:56 smithi070 ceph-mon[32820]: pgmap v324: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 224 op/s 2024-04-25T18:56:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:56 smithi114 ceph-mon[32988]: pgmap v324: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.5 MiB/s wr, 224 op/s 2024-04-25T18:56:57.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 333107 3.47 MB/sec execute 416 sec latency 20.383 ms 2024-04-25T18:56:58.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 333894 3.48 MB/sec execute 417 sec latency 15.310 ms 2024-04-25T18:56:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:58 smithi063 ceph-mon[28006]: pgmap v325: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 153 op/s 2024-04-25T18:56:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:58 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:58 smithi070 ceph-mon[32820]: pgmap v325: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 153 op/s 2024-04-25T18:56:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:58 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:58 smithi114 ceph-mon[32988]: pgmap v325: 97 pgs: 97 active+clean; 100 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.0 MiB/s wr, 153 op/s 2024-04-25T18:56:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:58 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:56:59.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 334900 3.48 MB/sec execute 418 sec latency 71.300 ms 2024-04-25T18:56:59.802 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:59 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.802 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:59 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.803 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:56:59 smithi114 ceph-mon[32988]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:56:59.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:59 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:59 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:56:59 smithi063 ceph-mon[28006]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:56:59.913 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:59 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.913 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:59 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:56:59.913 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:56:59 smithi070 ceph-mon[32820]: pgmap v326: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.5 MiB/s wr, 220 op/s 2024-04-25T18:57:00.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 336321 3.49 MB/sec execute 419 sec latency 20.560 ms 2024-04-25T18:57:01.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 337183 3.50 MB/sec execute 420 sec latency 42.104 ms 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:01 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:01 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:01.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:01 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:02.504 INFO:tasks.workunit.client.0.smithi063.stdout: 1 337766 3.49 MB/sec execute 421 sec latency 21.350 ms 2024-04-25T18:57:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:02 smithi114 ceph-mon[32988]: pgmap v327: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:02 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:57:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:02 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:02 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:57:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:02 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:02 smithi063 ceph-mon[28006]: pgmap v327: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:02 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:57:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:02 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:02 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:57:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:02 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:02 smithi070 ceph-mon[32820]: pgmap v327: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:02 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T18:57:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:02 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:02 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T18:57:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:02 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T18:57:03.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 338511 3.50 MB/sec execute 422 sec latency 45.568 ms 2024-04-25T18:57:03.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:03.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:03.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:57:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:57:04.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 339179 3.49 MB/sec execute 423 sec latency 464.437 ms 2024-04-25T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:04 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:04 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:57:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:04 smithi114 ceph-mon[32988]: pgmap v328: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:04 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:57:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:04 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:57:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:04 smithi063 ceph-mon[28006]: pgmap v328: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:04.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:04 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T18:57:04.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:04 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T18:57:04.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:04 smithi070 ceph-mon[32820]: pgmap v328: 97 pgs: 97 active+clean; 105 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 -- 172.21.15.63:0/1736311264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50109470 msgr2=0x7f9d50111040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 --2- 172.21.15.63:0/1736311264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50109470 0x7f9d50111040 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9d40009860 tx=0x7f9d4002a340 comp rx=0 tx=0).stop 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 -- 172.21.15.63:0/1736311264 shutdown_connections 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 --2- 172.21.15.63:0/1736311264 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d50109470 0x7f9d50111040 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 --2- 172.21.15.63:0/1736311264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50104d50 0x7f9d50108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.066 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 --2- 172.21.15.63:0/1736311264 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d501043a0 0x7f9d50104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 -- 172.21.15.63:0/1736311264 >> 172.21.15.63:0/1736311264 conn(0x7f9d500ffc40 msgr2=0x7f9d50102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 -- 172.21.15.63:0/1736311264 shutdown_connections 2024-04-25T18:57:05.067 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.064+0000 7f9d58200640 1 -- 172.21.15.63:0/1736311264 wait complete. 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.065+0000 7f9d58200640 1 Processor -- start 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 -- start start 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 0x7f9d501a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 0x7f9d501a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 0x7f9d5019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501143e0 con 0x7f9d50109470 2024-04-25T18:57:05.069 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50114260 con 0x7f9d501043a0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d58200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d50114560 con 0x7f9d5019a6a0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d54e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 0x7f9d501a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d55800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 0x7f9d501a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d56200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 0x7f9d5019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d54e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 0x7f9d501a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52072/0 (socket says 172.21.15.63:52072) 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.066+0000 7f9d54e00640 1 -- 172.21.15.63:0/3241695673 learned_addr learned my addr 172.21.15.63:0/3241695673 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 -- 172.21.15.63:0/3241695673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 msgr2=0x7f9d501a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 0x7f9d501a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 -- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 msgr2=0x7f9d501a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 0x7f9d501a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 -- 172.21.15.63:0/3241695673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5019b2a0 con 0x7f9d5019a6a0 2024-04-25T18:57:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d55800640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 0x7f9d501a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d56200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 0x7f9d5019ab20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002a860 tx=0x7f9d40032d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.067+0000 7f9d4e000640 1 -- 172.21.15.63:0/3241695673 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40037040 con 0x7f9d5019a6a0 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.068+0000 7f9d4e000640 1 -- 172.21.15.63:0/3241695673 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40004050 con 0x7f9d5019a6a0 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.068+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5019b530 con 0x7f9d5019a6a0 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.068+0000 7f9d4e000640 1 -- 172.21.15.63:0/3241695673 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40005020 con 0x7f9d5019a6a0 2024-04-25T18:57:05.071 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.068+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d501a7660 con 0x7f9d5019a6a0 2024-04-25T18:57:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.071+0000 7f9d4e000640 1 -- 172.21.15.63:0/3241695673 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9d40043020 con 0x7f9d5019a6a0 2024-04-25T18:57:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.071+0000 7f9d5af87740 1 Processor -- start 2024-04-25T18:57:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.071+0000 7f9d5af87740 1 -- start start 2024-04-25T18:57:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbc140 0x5603fbbc64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbc6a00 0x5603fbbc8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.074 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5603fbb21900 con 0x5603fbbbc140 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603fbac7ef0 con 0x5603fbbc6a00 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d5af87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603fbb1d8c0 con 0x5603fbbbb820 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d54e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d54e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56894/0 (socket says 172.21.15.63:56894) 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d54e00640 1 -- 172.21.15.63:0/2884851447 learned_addr learned my addr 172.21.15.63:0/2884851447 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d55800640 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbc140 0x5603fbbc64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d54e00640 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbc6a00 0x5603fbbc8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.072+0000 7f9d54e00640 1 -- 172.21.15.63:0/2884851447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbc6a00 msgr2=0x5603fbbc8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d54e00640 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbc6a00 0x5603fbbc8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d54e00640 1 -- 172.21.15.63:0/2884851447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbc140 msgr2=0x5603fbbc64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d54e00640 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbc140 0x5603fbbc64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d54e00640 1 -- 172.21.15.63:0/2884851447 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603fb834210 con 0x5603fbbbb820 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d54e00640 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5603fbbb8ec0 tx=0x7f9d3c0097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3ea1f266fa0cc6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.075 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d46a00640 1 -- 172.21.15.63:0/2884851447 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c013db0 con 0x5603fbbbb820 2024-04-25T18:57:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d46a00640 1 -- 172.21.15.63:0/2884851447 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c0054d0 con 0x5603fbbbb820 2024-04-25T18:57:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d46a00640 1 -- 172.21.15.63:0/2884851447 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c004df0 con 0x5603fbbbb820 2024-04-25T18:57:05.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.073+0000 7f9d4e000640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 0x7f9d340a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d4e000640 1 -- 172.21.15.63:0/3241695673 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 8631+0+0 (secure 0 0 0) 0x7f9d400efac0 con 0x7f9d5019a6a0 2024-04-25T18:57:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d55800640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 0x7f9d340a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.079 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 -- 172.21.15.63:0/2884851447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 msgr2=0x5603fbbbbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5603fbbb8ec0 tx=0x7f9d3c0097d0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 -- 172.21.15.63:0/2884851447 shutdown_connections 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbc6a00 0x5603fbbc8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbc140 0x5603fbbc64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 --2- 172.21.15.63:0/2884851447 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbbbc00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 -- 172.21.15.63:0/2884851447 >> 172.21.15.63:0/2884851447 conn(0x5603fbbb7d30 msgr2=0x5603fbbb8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 -- 172.21.15.63:0/2884851447 shutdown_connections 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d5af87740 1 -- 172.21.15.63:0/2884851447 wait complete. 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.074+0000 7f9d55800640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 0x7f9d340a6300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x5603fbbbcdc0 tx=0x7f9d48009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 -- start start 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 Processor -- start 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbc3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.080 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbc4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.075+0000 7f9d5af87740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5603fbbc9630 con 0x5603fbbbb820 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d5af87740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5603fbadb310 con 0x5603fbbbc140 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d5af87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5603fbbc97b0 con 0x5603fbbc6a00 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d5af87740 10 client.0 _ll_register_callbacks cb 0x5603fbaa9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d55800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbc3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbc4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d54e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d55800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbc3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52080/0 (socket says 172.21.15.63:52080) 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d55800640 1 -- 172.21.15.63:0/3081674555 learned_addr learned my addr 172.21.15.63:0/3081674555 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 -- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 msgr2=0x5603fbbc3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 -- 172.21.15.63:0/3081674555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 msgr2=0x5603fbbc3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbc3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d56200640 1 -- 172.21.15.63:0/3081674555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603fb900620 con 0x5603fbbc6a00 2024-04-25T18:57:05.081 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.076+0000 7f9d54e00640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d56200640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbc4520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9d400ed250 tx=0x7f9d400bb400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a14c9d4cd4ac363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d400bbaa0 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8631+0+0 (secure 0 0 0) 0x7f9d400b8110 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d3c0054d0 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9d400bfe50 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d5af87740 10 client.26473 fetch_fsmap learned FSMap version 42 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5603fbbc9630 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d3c0054d0 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f9d40030b50 con 0x5603fbbc6a00 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.077+0000 7f9d5af87740 10 client.26473 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d5af87740 10 client.26473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d5af87740 20 client.26473 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbbd6e50 0x5603fbbd9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d5af87740 4 client.26473 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603fbb358b0 con 0x5603fbbd6e50 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d55800640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbbd6e50 0x5603fbbd9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d55800640 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbbd6e50 0x5603fbbd9290 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2606eb42889011 server_cookie=a42ac182d5646cce in_seq=0 out_seq=0 2024-04-25T18:57:05.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.078+0000 7f9d3b400640 10 client.26473 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.079+0000 7f9d3b400640 1 -- 172.21.15.63:0/3081674555 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5603fbb358b0 con 0x5603fbbd6e50 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.079+0000 7f9d3b400640 10 client.26473 handle_command_reply: tid=0 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.079+0000 7f9d5af87740 1 client.26473 shutdown 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 msgr2=0x5603fbbc4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbc4520 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9d400ed250 tx=0x7f9d400bb400 comp rx=0 tx=0).stop 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 shutdown_connections 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbbd6e50 0x5603fbbd9290 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.084 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbc4520 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 --2- 172.21.15.63:0/3081674555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbc3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d3b400640 0 client.26473 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 >> 172.21.15.63:0/3081674555 conn(0x5603fbbb7d30 msgr2=0x5603fbbbdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 shutdown_connections 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 1 -- 172.21.15.63:0/3081674555 wait complete. 2024-04-25T18:57:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.080+0000 7f9d5af87740 20 client.26473 trim_cache size 0 max 16384 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 Processor -- start 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 -- start start 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbbd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbdeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501b2750 con 0x5603fbbbb820 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501aff60 con 0x5603fbbbc140 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d5af87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501addd0 con 0x5603fbbc6a00 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d55800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d55800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52082/0 (socket says 172.21.15.63:52082) 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d55800640 1 -- 172.21.15.63:0/18442678 learned_addr learned my addr 172.21.15.63:0/18442678 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:05.099 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d56200640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbdeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.097+0000 7f9d54e00640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbbd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 -- 172.21.15.63:0/18442678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 msgr2=0x5603fbbbd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbbd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 -- 172.21.15.63:0/18442678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 msgr2=0x5603fbbd24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 -- 172.21.15.63:0/18442678 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603fbbd9910 con 0x5603fbbc6a00 2024-04-25T18:57:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d55800640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d56200640 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbdeba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d400b8470 tx=0x7f9d400b9fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2554ad231f4e169 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d46000640 1 -- 172.21.15.63:0/18442678 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d400bb870 con 0x5603fbbc6a00 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d46000640 1 -- 172.21.15.63:0/18442678 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d40004490 con 0x5603fbbc6a00 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.098+0000 7f9d46000640 1 -- 172.21.15.63:0/18442678 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d400b7d40 con 0x5603fbbc6a00 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 -- 172.21.15.63:0/18442678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 msgr2=0x5603fbbdeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbdeba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9d400b8470 tx=0x7f9d400b9fb0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 -- 172.21.15.63:0/18442678 shutdown_connections 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbbd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbc6a00 0x5603fbbdeba0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 --2- 172.21.15.63:0/18442678 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbbb820 0x5603fbbd24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 -- 172.21.15.63:0/18442678 >> 172.21.15.63:0/18442678 conn(0x5603fbbb7d30 msgr2=0x5603fbbbd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 -- 172.21.15.63:0/18442678 shutdown_connections 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.099+0000 7f9d5af87740 1 -- 172.21.15.63:0/18442678 wait complete. 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 -- start start 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 Processor -- start 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 0x5603fbbc1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501ae420 con 0x5603fbbc6a00 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501af890 con 0x5603fbbbc140 2024-04-25T18:57:05.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9d501aeaf0 con 0x5603fbbbb820 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d5af87740 10 client.0 _ll_register_callbacks cb 0x7f9d40005c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d55800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.100+0000 7f9d55800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56916/0 (socket says 172.21.15.63:56916) 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 -- 172.21.15.63:0/1781251935 learned_addr learned my addr 172.21.15.63:0/1781251935 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d56200640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 0x5603fbbc1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d54e00640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 -- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 msgr2=0x5603fbbc0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 -- 172.21.15.63:0/1781251935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 msgr2=0x5603fbbc1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 0x5603fbbc1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 -- 172.21.15.63:0/1781251935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603fbc3c440 con 0x5603fbbbb820 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d56200640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 0x5603fbbc1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d55800640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800fd60 tx=0x7f9d48004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1377d0af706c76da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:05.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d3b400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4801f920 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5603fbb2f320 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.101+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 8631+0+0 (secure 0 0 0) 0x7f9d4801fac0 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f9d4801fcc0 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 10 client.26479 fetch_fsmap learned FSMap version 42 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5603fbc3d790 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5603fbb2f320 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 42) ==== 8354+0+0 (secure 0 0 0) 0x7f9d4803b790 con 0x5603fbbbb820 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 10 client.26479 fetch_fsmap finished waiting for FSMap version 42 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 20 client.26479 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbc3def0 0x5603fbc40310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 4 client.26479 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:57:05.106 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5603fbb29190 con 0x5603fbc3def0 2024-04-25T18:57:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.102+0000 7f9d54e00640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbc3def0 0x5603fbc40310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.103+0000 7f9d54e00640 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbc3def0 0x5603fbc40310 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9583f425be19bf6 server_cookie=25758cf77b1e0fc4 in_seq=0 out_seq=0 2024-04-25T18:57:05.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.103+0000 7f9d3b400640 10 client.26479 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d3b400640 1 -- 172.21.15.63:0/1781251935 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5603fbb29190 con 0x5603fbc3def0 2024-04-25T18:57:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d3b400640 10 client.26479 handle_command_reply: tid=0 2024-04-25T18:57:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 client.26479 shutdown 2024-04-25T18:57:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 msgr2=0x5603fbbc0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9d4800fd60 tx=0x7f9d48004a40 comp rx=0 tx=0).stop 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 shutdown_connections 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5603fbc3def0 0x5603fbc40310 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5603fbbbc140 0x5603fbbc0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603fbbc6a00 0x5603fbbc1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d3b400640 0 client.26479 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 --2- 172.21.15.63:0/1781251935 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5603fbbbb820 0x5603fbbc0790 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 >> 172.21.15.63:0/1781251935 conn(0x5603fbbb7d30 msgr2=0x5603fbbc42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 shutdown_connections 2024-04-25T18:57:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 1 -- 172.21.15.63:0/1781251935 wait complete. 2024-04-25T18:57:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.130+0000 7f9d5af87740 20 client.26479 trim_cache size 0 max 16384 2024-04-25T18:57:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:57:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:57:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:57:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:57:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 msgr2=0x7f9d340a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.137 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 0x7f9d340a6300 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x5603fbbbcdc0 tx=0x7f9d48009500 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 msgr2=0x7f9d5019ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 0x7f9d5019ab20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9d4002a860 tx=0x7f9d40032d20 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 shutdown_connections 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9d340a3e40 0x7f9d340a6300 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9d5019a6a0 0x7f9d5019ab20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d50109470 0x7f9d501a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 --2- 172.21.15.63:0/3241695673 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d501043a0 0x7f9d501a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:05.138 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 >> 172.21.15.63:0/3241695673 conn(0x7f9d500ffc40 msgr2=0x7f9d50101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:05.139 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.131+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 shutdown_connections 2024-04-25T18:57:05.139 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:05.132+0000 7f9d58200640 1 -- 172.21.15.63:0/3241695673 wait complete. 2024-04-25T18:57:05.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 339179 3.48 MB/sec execute 424 sec latency 1464.583 ms 2024-04-25T18:57:05.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:05 smithi070 ceph-mon[32820]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T18:57:05.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:05 smithi070 ceph-mon[32820]: pgmap v330: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:06.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:05 smithi114 ceph-mon[32988]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T18:57:06.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:05 smithi114 ceph-mon[32988]: pgmap v330: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:06.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:05 smithi063 ceph-mon[28006]: osdmap e108: 12 total, 12 up, 12 in 2024-04-25T18:57:06.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:05 smithi063 ceph-mon[28006]: pgmap v330: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:06.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 339484 3.48 MB/sec execute 425 sec latency 2134.031 ms 2024-04-25T18:57:07.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 340774 3.50 MB/sec execute 426 sec latency 16.493 ms 2024-04-25T18:57:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:07 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:08.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:07 smithi114 ceph-mon[32988]: pgmap v331: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:08.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:07 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:08.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:07 smithi063 ceph-mon[28006]: pgmap v331: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:07 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:08.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:07 smithi070 ceph-mon[32820]: pgmap v331: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.2 MiB/s wr, 199 op/s 2024-04-25T18:57:08.505 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.49 MB/sec execute 427 sec latency 456.283 ms 2024-04-25T18:57:09.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.48 MB/sec execute 428 sec latency 1456.431 ms 2024-04-25T18:57:10.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.47 MB/sec execute 429 sec latency 2456.592 ms 2024-04-25T18:57:10.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:10 smithi114 ceph-mon[32988]: pgmap v332: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:10 smithi063 ceph-mon[28006]: pgmap v332: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:10 smithi070 ceph-mon[32820]: pgmap v332: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:11.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.46 MB/sec execute 430 sec latency 3456.738 ms 2024-04-25T18:57:12.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.46 MB/sec execute 431 sec latency 4456.839 ms 2024-04-25T18:57:12.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:12 smithi114 ceph-mon[32988]: pgmap v333: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:12.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:12 smithi063 ceph-mon[28006]: pgmap v333: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:12 smithi070 ceph-mon[32820]: pgmap v333: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:13.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.45 MB/sec execute 432 sec latency 5456.972 ms 2024-04-25T18:57:14.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.44 MB/sec execute 433 sec latency 6457.089 ms 2024-04-25T18:57:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:14 smithi114 ceph-mon[32988]: pgmap v334: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:14.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:14 smithi114 ceph-mon[32988]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T18:57:14.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:14 smithi063 ceph-mon[28006]: pgmap v334: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:14.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:14 smithi063 ceph-mon[28006]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T18:57:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:14 smithi070 ceph-mon[32820]: pgmap v334: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.0 MiB/s wr, 170 op/s 2024-04-25T18:57:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:14 smithi070 ceph-mon[32820]: osdmap e109: 12 total, 12 up, 12 in 2024-04-25T18:57:15.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.43 MB/sec execute 434 sec latency 7457.265 ms 2024-04-25T18:57:16.506 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.42 MB/sec execute 435 sec latency 8457.407 ms 2024-04-25T18:57:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:16 smithi114 ceph-mon[32988]: pgmap v336: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:16 smithi063 ceph-mon[28006]: pgmap v336: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:16 smithi070 ceph-mon[32820]: pgmap v336: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:17.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.42 MB/sec execute 436 sec latency 9457.501 ms 2024-04-25T18:57:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 -- 172.21.15.63:0/1711827967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c102980 msgr2=0x7f5e2c1069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 --2- 172.21.15.63:0/1711827967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c102980 0x7f5e2c1069c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009900 tx=0x7f5e1802a410 comp rx=0 tx=0).stop 2024-04-25T18:57:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 -- 172.21.15.63:0/1711827967 shutdown_connections 2024-04-25T18:57:18.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 --2- 172.21.15.63:0/1711827967 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 --2- 172.21.15.63:0/1711827967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c102980 0x7f5e2c1069c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 --2- 172.21.15.63:0/1711827967 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c101fd0 0x7f5e2c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 -- 172.21.15.63:0/1711827967 >> 172.21.15.63:0/1711827967 conn(0x7f5e2c0fd890 msgr2=0x7f5e2c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 -- 172.21.15.63:0/1711827967 shutdown_connections 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.283+0000 7f5e30e00640 1 -- 172.21.15.63:0/1711827967 wait complete. 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.284+0000 7f5e30e00640 1 Processor -- start 2024-04-25T18:57:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.284+0000 7f5e30e00640 1 -- start start 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c101fd0 0x7f5e2c1a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c19a5a0 0x7f5e2c19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c111fe0 con 0x7f5e2c1070a0 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c111e60 con 0x7f5e2c19a5a0 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e30e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c112160 con 0x7f5e2c101fd0 2024-04-25T18:57:18.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56446/0 (socket says 172.21.15.63:56446) 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 -- 172.21.15.63:0/655365348 learned_addr learned my addr 172.21.15.63:0/655365348 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2b400640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c101fd0 0x7f5e2c1a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2be00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c19a5a0 0x7f5e2c19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 -- 172.21.15.63:0/655365348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c101fd0 msgr2=0x7f5e2c1a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c101fd0 0x7f5e2c1a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 -- 172.21.15.63:0/655365348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c19a5a0 msgr2=0x7f5e2c19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c19a5a0 0x7f5e2c19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.285+0000 7f5e2aa00640 1 -- 172.21.15.63:0/655365348 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c19b0c0 con 0x7f5e2c1070a0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5e18032fd0 tx=0x7f5e18031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e23e00640 1 -- 172.21.15.63:0/655365348 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1803f030 con 0x7f5e2c1070a0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c19b350 con 0x7f5e2c1070a0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e23e00640 1 -- 172.21.15.63:0/655365348 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e18004490 con 0x7f5e2c1070a0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e23e00640 1 -- 172.21.15.63:0/655365348 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18035070 con 0x7f5e2c1070a0 2024-04-25T18:57:18.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.286+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c1a7540 con 0x7f5e2c1070a0 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.291+0000 7f5e351b2740 1 Processor -- start 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.291+0000 7f5e351b2740 1 -- start start 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.291+0000 7f5e23e00640 1 -- 172.21.15.63:0/655365348 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5e18042030 con 0x7f5e2c1070a0 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a769b0 0x5584d4a76d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a83f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5584d49dc900 con 0x5584d4a81b90 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584d4982ef0 con 0x5584d4a772d0 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e351b2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584d49d88c0 con 0x5584d4a769b0 2024-04-25T18:57:18.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a83f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38028/0 (socket says 172.21.15.63:38028) 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a769b0 0x5584d4a76d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2b400640 1 -- 172.21.15.63:0/3099305109 learned_addr learned my addr 172.21.15.63:0/3099305109 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.292+0000 7f5e2b400640 1 -- 172.21.15.63:0/3099305109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a769b0 msgr2=0x5584d4a76d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2b400640 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a769b0 0x5584d4a76d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2b400640 1 -- 172.21.15.63:0/3099305109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 msgr2=0x5584d4a83f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2b400640 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a83f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2b400640 1 -- 172.21.15.63:0/3099305109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d46ef210 con 0x5584d4a772d0 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a83f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:18.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e2b400640 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009cf0 tx=0x7f5e1c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a1679e24b0203c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e20c00640 1 -- 172.21.15.63:0/3099305109 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c01e070 con 0x5584d4a772d0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e20c00640 1 -- 172.21.15.63:0/3099305109 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1c019ae0 con 0x5584d4a772d0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e20c00640 1 -- 172.21.15.63:0/3099305109 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00e970 con 0x5584d4a772d0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.293+0000 7f5e23e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 0x7f5e100a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e23e00640 1 -- 172.21.15.63:0/655365348 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f5e180f0b60 con 0x7f5e2c1070a0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e2be00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 0x7f5e100a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 -- 172.21.15.63:0/3099305109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 msgr2=0x5584d4a81650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009cf0 tx=0x7f5e1c011810 comp rx=0 tx=0).stop 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 -- 172.21.15.63:0/3099305109 shutdown_connections 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a772d0 0x5584d4a81650 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a83f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 --2- 172.21.15.63:0/3099305109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a769b0 0x5584d4a76d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 -- 172.21.15.63:0/3099305109 >> 172.21.15.63:0/3099305109 conn(0x5584d4a72ec0 msgr2=0x5584d4a732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 -- 172.21.15.63:0/3099305109 shutdown_connections 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e2be00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 0x7f5e100a61d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e24005fd0 tx=0x7f5e24009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.294+0000 7f5e351b2740 1 -- 172.21.15.63:0/3099305109 wait complete. 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.295+0000 7f5e351b2740 1 -- start start 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.295+0000 7f5e351b2740 1 Processor -- start 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a7eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a7f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5584d4a847c0 con 0x5584d4a81b90 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584d4996310 con 0x5584d4a769b0 2024-04-25T18:57:18.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584d4a84940 con 0x5584d4a772d0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e351b2740 10 client.0 _ll_register_callbacks cb 0x5584d4964cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a7eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56470/0 (socket says 172.21.15.63:56470) 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2be00640 1 -- 172.21.15.63:0/381007242 learned_addr learned my addr 172.21.15.63:0/381007242 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a7f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 -- 172.21.15.63:0/381007242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 msgr2=0x5584d4a7f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a7f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 -- 172.21.15.63:0/381007242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 msgr2=0x5584d4a7f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2b400640 1 -- 172.21.15.63:0/381007242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d47bb620 con 0x5584d4a769b0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.296+0000 7f5e2be00640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e2b400640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a7eaf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c01c040 tx=0x7f5e1c019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49cfa614983af43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e15600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c01e050 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f5e1c005240 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e1c019ae0 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5e1c033050 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e351b2740 10 client.26471 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.297+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5584d4a909e0 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e1c019ae0 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f5e1c004b70 con 0x5584d4a769b0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 10 client.26471 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 10 client.26471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 20 client.26471 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4a91de0 0x5584d4a941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 4 client.26471 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584d49f08b0 con 0x5584d4a91de0 2024-04-25T18:57:18.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.298+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4a91de0 0x5584d4a941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.299+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4a91de0 0x5584d4a941d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42825c07ef02750 server_cookie=d4cdf03ae5790ce in_seq=0 out_seq=0 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.299+0000 7f5e15600640 10 client.26471 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e15600640 1 -- 172.21.15.63:0/381007242 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584d49f08b0 con 0x5584d4a91de0 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e15600640 10 client.26471 handle_command_reply: tid=0 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 client.26471 shutdown 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 msgr2=0x5584d4a7eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a7eaf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c01c040 tx=0x7f5e1c019660 comp rx=0 tx=0).stop 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 shutdown_connections 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4a91de0 0x5584d4a941d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4a7f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a7f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 --2- 172.21.15.63:0/381007242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a7eaf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 >> 172.21.15.63:0/381007242 conn(0x5584d4a72ec0 msgr2=0x5584d4a78bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e15600640 0 client.26471 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 shutdown_connections 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 1 -- 172.21.15.63:0/381007242 wait complete. 2024-04-25T18:57:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.300+0000 7f5e351b2740 20 client.26471 trim_cache size 0 max 16384 2024-04-25T18:57:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 Processor -- start 2024-04-25T18:57:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 -- start start 2024-04-25T18:57:18.318 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a8d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.319 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.319 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a81b90 0x5584d4a99980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.319 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1b2660 con 0x5584d4a772d0 2024-04-25T18:57:18.319 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1afe70 con 0x5584d4a769b0 2024-04-25T18:57:18.319 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.317+0000 7f5e351b2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1adce0 con 0x5584d4a81b90 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56486/0 (socket says 172.21.15.63:56486) 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 -- 172.21.15.63:0/1197794207 learned_addr learned my addr 172.21.15.63:0/1197794207 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2b400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a8d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2be00640 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a81b90 0x5584d4a99980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 -- 172.21.15.63:0/1197794207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a81b90 msgr2=0x5584d4a99980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.321 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a81b90 0x5584d4a99980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 -- 172.21.15.63:0/1197794207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 msgr2=0x5584d4a8d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a8d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 -- 172.21.15.63:0/1197794207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584d4a77100 con 0x5584d4a772d0 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2b400640 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a8d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.318+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5e180b9540 tx=0x7f5e180dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22aab71d614af7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.319+0000 7f5e17e00640 1 -- 172.21.15.63:0/1197794207 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18049040 con 0x5584d4a772d0 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.319+0000 7f5e17e00640 1 -- 172.21.15.63:0/1197794207 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e180bcb70 con 0x5584d4a772d0 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.319+0000 7f5e17e00640 1 -- 172.21.15.63:0/1197794207 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e180bae70 con 0x5584d4a772d0 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.319+0000 7f5e351b2740 1 -- 172.21.15.63:0/1197794207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 msgr2=0x5584d4a78a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.319+0000 7f5e351b2740 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5e180b9540 tx=0x7f5e180dd040 comp rx=0 tx=0).stop 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 -- 172.21.15.63:0/1197794207 shutdown_connections 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a81b90 0x5584d4a99980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a772d0 0x5584d4a78a70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.322 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 --2- 172.21.15.63:0/1197794207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a8d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 -- 172.21.15.63:0/1197794207 >> 172.21.15.63:0/1197794207 conn(0x5584d4a72ec0 msgr2=0x5584d4a78360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 -- 172.21.15.63:0/1197794207 shutdown_connections 2024-04-25T18:57:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 -- 172.21.15.63:0/1197794207 wait complete. 2024-04-25T18:57:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 -- start start 2024-04-25T18:57:18.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.320+0000 7f5e351b2740 1 Processor -- start 2024-04-25T18:57:18.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a86df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4af01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1ae330 con 0x5584d4a81b90 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1af7a0 con 0x5584d4a769b0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c1aea00 con 0x5584d4a772d0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e351b2740 10 client.0 _ll_register_callbacks cb 0x7f5e24009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4af01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:34400/0 (socket says 172.21.15.63:34400) 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 -- 172.21.15.63:0/2989127119 learned_addr learned my addr 172.21.15.63:0/2989127119 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2b400640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a86df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 -- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 msgr2=0x5584d4a86df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a86df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 -- 172.21.15.63:0/2989127119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 msgr2=0x5584d4af01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4af01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2aa00640 1 -- 172.21.15.63:0/2989127119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584d4af07b0 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.321+0000 7f5e2be00640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4af01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e2aa00640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5e180bc870 tx=0x7f5e18009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132d05c914f9dc6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e15600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18049050 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e180bcb70 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f5e180bae70 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f5e180bc6b0 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e351b2740 10 client.26497 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.322+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5584d4af7b40 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e180bcb70 con 0x5584d4a772d0 2024-04-25T18:57:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f5e180b91e0 con 0x5584d4a772d0 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 10 client.26497 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 10 client.26497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 20 client.26497 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4af82a0 0x5584d4afa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 4 client.26497 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584d49ea320 con 0x5584d4af82a0 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.323+0000 7f5e2b400640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4af82a0 0x5584d4afa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.324+0000 7f5e2b400640 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4af82a0 0x5584d4afa6c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285d3d47614ff9df server_cookie=a1e8777ec25af955 in_seq=0 out_seq=0 2024-04-25T18:57:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.324+0000 7f5e15600640 10 client.26497 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.350+0000 7f5e15600640 1 -- 172.21.15.63:0/2989127119 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584d49ea320 con 0x5584d4af82a0 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.350+0000 7f5e15600640 10 client.26497 handle_command_reply: tid=0 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.350+0000 7f5e351b2740 1 client.26497 shutdown 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.350+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 msgr2=0x5584d4a87330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.350+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5e180bc870 tx=0x7f5e18009680 comp rx=0 tx=0).stop 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 shutdown_connections 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584d4af82a0 0x5584d4afa6c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584d4a81b90 0x5584d4af01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584d4a772d0 0x5584d4a87330 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 --2- 172.21.15.63:0/2989127119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584d4a769b0 0x5584d4a86df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 >> 172.21.15.63:0/2989127119 conn(0x5584d4a72ec0 msgr2=0x5584d4a78bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e15600640 0 client.26497 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:18.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 shutdown_connections 2024-04-25T18:57:18.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 1 -- 172.21.15.63:0/2989127119 wait complete. 2024-04-25T18:57:18.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.351+0000 7f5e351b2740 20 client.26497 trim_cache size 0 max 16384 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 msgr2=0x7f5e100a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 0x7f5e100a61d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e24005fd0 tx=0x7f5e24009500 comp rx=0 tx=0).stop 2024-04-25T18:57:18.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 msgr2=0x7f5e2c1a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5e18032fd0 tx=0x7f5e18031a20 comp rx=0 tx=0).stop 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 shutdown_connections 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5e100a3d10 0x7f5e100a61d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5e2c19a5a0 0x7f5e2c19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e2c1070a0 0x7f5e2c1a0930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 --2- 172.21.15.63:0/655365348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5e2c101fd0 0x7f5e2c1a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.352+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 >> 172.21.15.63:0/655365348 conn(0x7f5e2c0fd890 msgr2=0x7f5e2c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.353+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 shutdown_connections 2024-04-25T18:57:18.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:18.353+0000 7f5e30e00640 1 -- 172.21.15.63:0/655365348 wait complete. 2024-04-25T18:57:18.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.41 MB/sec execute 437 sec latency 10457.648 ms 2024-04-25T18:57:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:18 smithi114 ceph-mon[32988]: pgmap v337: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:18 smithi063 ceph-mon[28006]: pgmap v337: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:18 smithi070 ceph-mon[32820]: pgmap v337: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 879 KiB/s wr, 69 op/s 2024-04-25T18:57:19.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.40 MB/sec execute 438 sec latency 11457.966 ms 2024-04-25T18:57:20.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341050 3.39 MB/sec execute 439 sec latency 12458.080 ms 2024-04-25T18:57:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:20 smithi114 ceph-mon[32988]: pgmap v338: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:20 smithi063 ceph-mon[28006]: pgmap v338: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:20 smithi070 ceph-mon[32820]: pgmap v338: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:21.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 341568 3.39 MB/sec execute 440 sec latency 12842.089 ms 2024-04-25T18:57:22.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 342916 3.39 MB/sec execute 441 sec latency 24.282 ms 2024-04-25T18:57:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:22 smithi114 ceph-mon[32988]: pgmap v339: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:22 smithi063 ceph-mon[28006]: pgmap v339: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:22 smithi070 ceph-mon[32820]: pgmap v339: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:23.507 INFO:tasks.workunit.client.0.smithi063.stdout: 1 344223 3.41 MB/sec execute 442 sec latency 13.989 ms 2024-04-25T18:57:24.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 344791 3.41 MB/sec execute 443 sec latency 30.126 ms 2024-04-25T18:57:24.566 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:24 smithi114 ceph-mon[32988]: pgmap v340: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:24 smithi063 ceph-mon[28006]: pgmap v340: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:24 smithi070 ceph-mon[32820]: pgmap v340: 97 pgs: 97 active+clean; 106 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 961 KiB/s wr, 50 op/s 2024-04-25T18:57:25.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 345599 3.41 MB/sec execute 444 sec latency 27.930 ms 2024-04-25T18:57:26.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 346940 3.42 MB/sec execute 445 sec latency 21.675 ms 2024-04-25T18:57:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:26 smithi114 ceph-mon[32988]: pgmap v341: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-25T18:57:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:26 smithi063 ceph-mon[28006]: pgmap v341: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-25T18:57:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:26 smithi070 ceph-mon[32820]: pgmap v341: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.3 MiB/s wr, 104 op/s 2024-04-25T18:57:27.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 348001 3.43 MB/sec execute 446 sec latency 17.456 ms 2024-04-25T18:57:28.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 348868 3.42 MB/sec execute 447 sec latency 21.119 ms 2024-04-25T18:57:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:28 smithi114 ceph-mon[32988]: pgmap v342: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 861 KiB/s wr, 82 op/s 2024-04-25T18:57:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:28 smithi063 ceph-mon[28006]: pgmap v342: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 861 KiB/s wr, 82 op/s 2024-04-25T18:57:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:28 smithi070 ceph-mon[32820]: pgmap v342: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 861 KiB/s wr, 82 op/s 2024-04-25T18:57:29.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 350145 3.43 MB/sec execute 448 sec latency 29.484 ms 2024-04-25T18:57:30.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351434 3.44 MB/sec execute 449 sec latency 18.845 ms 2024-04-25T18:57:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:30 smithi114 ceph-mon[32988]: pgmap v343: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-25T18:57:30.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:30 smithi063 ceph-mon[28006]: pgmap v343: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-25T18:57:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:30 smithi070 ceph-mon[32820]: pgmap v343: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 162 op/s 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 -- 172.21.15.63:0/4026894431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6254103870 msgr2=0x7f625410bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 --2- 172.21.15.63:0/4026894431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6254103870 0x7f625410bd50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6240009900 tx=0x7f624002a480 comp rx=0 tx=0).stop 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 -- 172.21.15.63:0/4026894431 shutdown_connections 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 --2- 172.21.15.63:0/4026894431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f625410c370 0x7f625410e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 --2- 172.21.15.63:0/4026894431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6254103870 0x7f625410bd50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 --2- 172.21.15.63:0/4026894431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f6254103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.500+0000 7f6253e00640 1 -- 172.21.15.63:0/4026894431 >> 172.21.15.63:0/4026894431 conn(0x7f62540fc770 msgr2=0x7f62540feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.501+0000 7f6253e00640 1 -- 172.21.15.63:0/4026894431 shutdown_connections 2024-04-25T18:57:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.501+0000 7f6253e00640 1 -- 172.21.15.63:0/4026894431 wait complete. 2024-04-25T18:57:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.502+0000 7f6253e00640 1 Processor -- start 2024-04-25T18:57:31.503 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.502+0000 7f6253e00640 1 -- start start 2024-04-25T18:57:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f62541985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6254103870 0x7f6254198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 0x7f625419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6254112030 con 0x7f6254103870 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6254111eb0 con 0x7f6254102f50 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62541121b0 con 0x7f625410c370 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6252a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f62541985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6252a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f62541985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34706/0 (socket says 172.21.15.63:34706) 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6252a00640 1 -- 172.21.15.63:0/1533999179 learned_addr learned my addr 172.21.15.63:0/1533999179 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6252000640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6254103870 0x7f6254198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.503+0000 7f6253400640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 0x7f625419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 -- 172.21.15.63:0/1533999179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 msgr2=0x7f62541985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f62541985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 -- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6254103870 msgr2=0x7f6254198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6254103870 0x7f6254198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 -- 172.21.15.63:0/1533999179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625419d5a0 con 0x7f625410c370 2024-04-25T18:57:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253400640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 0x7f625419ce70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f624c00da10 tx=0x7f624c00ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f624b400640 1 -- 172.21.15.63:0/1533999179 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c00bd70 con 0x7f625410c370 2024-04-25T18:57:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f624b400640 1 -- 172.21.15.63:0/1533999179 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f624c003e20 con 0x7f625410c370 2024-04-25T18:57:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625419d890 con 0x7f625410c370 2024-04-25T18:57:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.504+0000 7f624b400640 1 -- 172.21.15.63:0/1533999179 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624c0106c0 con 0x7f625410c370 2024-04-25T18:57:31.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.505+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62541a5120 con 0x7f625410c370 2024-04-25T18:57:31.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.44 MB/sec execute 450 sec latency 382.872 ms 2024-04-25T18:57:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.507+0000 7f624b400640 1 -- 172.21.15.63:0/1533999179 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f624c004270 con 0x7f625410c370 2024-04-25T18:57:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 Processor -- start 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 -- start start 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa34ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa2a2900 con 0x55a7fa34bfa0 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa248ef0 con 0x55a7fa340dc0 2024-04-25T18:57:31.511 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.508+0000 7f625c5cd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa29e8c0 con 0x55a7fa3416e0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34722/0 (socket says 172.21.15.63:34722) 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 -- 172.21.15.63:0/3762322537 learned_addr learned my addr 172.21.15.63:0/3762322537 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa34ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 -- 172.21.15.63:0/3762322537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 msgr2=0x55a7fa34ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa34ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 -- 172.21.15.63:0/3762322537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 msgr2=0x55a7fa34e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252000640 1 -- 172.21.15.63:0/3762322537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7f9fb5210 con 0x55a7fa340dc0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f6252a00640 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa34ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.509+0000 7f624b400640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 0x7f62380a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f6252000640 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f62400096c0 tx=0x7f6240005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25e56af4da0cb258 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f624b400640 1 -- 172.21.15.63:0/1533999179 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f624c0d0b90 con 0x7f625410c370 2024-04-25T18:57:31.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f6252a00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 0x7f62380a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f623f400640 1 -- 172.21.15.63:0/3762322537 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240035050 con 0x55a7fa340dc0 2024-04-25T18:57:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f623f400640 1 -- 172.21.15.63:0/3762322537 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6240003de0 con 0x55a7fa340dc0 2024-04-25T18:57:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.510+0000 7f623f400640 1 -- 172.21.15.63:0/3762322537 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240031b60 con 0x55a7fa340dc0 2024-04-25T18:57:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- 172.21.15.63:0/3762322537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 msgr2=0x55a7fa3411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f62400096c0 tx=0x7f6240005a20 comp rx=0 tx=0).stop 2024-04-25T18:57:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f6252a00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 0x7f62380a61d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55a7fa342360 tx=0x7f6244005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- 172.21.15.63:0/3762322537 shutdown_connections 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa34ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3762322537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa3411a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- 172.21.15.63:0/3762322537 >> 172.21.15.63:0/3762322537 conn(0x55a7fa33d2d0 msgr2=0x55a7fa33d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- 172.21.15.63:0/3762322537 shutdown_connections 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- 172.21.15.63:0/3762322537 wait complete. 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.511+0000 7f625c5cd740 1 -- start start 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 Processor -- start 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa348ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa349420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa34eb70 con 0x55a7fa34bfa0 2024-04-25T18:57:31.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa25c310 con 0x55a7fa340dc0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a7fa34ecf0 con 0x55a7fa3416e0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f625c5cd740 10 client.0 _ll_register_callbacks cb 0x55a7fa22acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f6252a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa348ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f6253400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f6253400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52498/0 (socket says 172.21.15.63:52498) 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.512+0000 7f6253400640 1 -- 172.21.15.63:0/1083843287 learned_addr learned my addr 172.21.15.63:0/1083843287 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa349420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 -- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 msgr2=0x55a7fa348ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa348ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 -- 172.21.15.63:0/1083843287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 msgr2=0x55a7fa349960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 -- 172.21.15.63:0/1083843287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fa081620 con 0x55a7fa3416e0 2024-04-25T18:57:31.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252a00640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa348ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6253400640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6252000640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa349420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f624002aa70 tx=0x7f6240005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c5dd5b7b77d4df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6233e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624004e020 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.513+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f62400316a0 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6240003de0 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f6240059050 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 10 client.26509 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a7fa34eb70 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6240003de0 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f624003b460 con 0x55a7fa3416e0 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 10 client.26509 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 10 client.26509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 20 client.26509 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:31.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa35bf40 0x55a7fa35e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 4 client.26509 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:57:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.514+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7fa2b68b0 con 0x55a7fa35bf40 2024-04-25T18:57:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.515+0000 7f6252a00640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa35bf40 0x55a7fa35e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.515+0000 7f6252a00640 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa35bf40 0x55a7fa35e330 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c4d1910a21e5731 server_cookie=45b7dc5d3cea97fe in_seq=0 out_seq=0 2024-04-25T18:57:31.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.515+0000 7f6233e00640 10 client.26509 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.516+0000 7f6233e00640 1 -- 172.21.15.63:0/1083843287 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7fa2b68b0 con 0x55a7fa35bf40 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.516+0000 7f6233e00640 10 client.26509 handle_command_reply: tid=0 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.516+0000 7f625c5cd740 1 client.26509 shutdown 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 msgr2=0x55a7fa349420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa349420 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f624002aa70 tx=0x7f6240005fd0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 shutdown_connections 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa35bf40 0x55a7fa35e330 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa349960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa349420 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 --2- 172.21.15.63:0/1083843287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa348ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 >> 172.21.15.63:0/1083843287 conn(0x55a7fa33d2d0 msgr2=0x55a7fa338020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f6233e00640 0 client.26509 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 shutdown_connections 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 1 -- 172.21.15.63:0/1083843287 wait complete. 2024-04-25T18:57:31.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.517+0000 7f625c5cd740 20 client.26509 trim_cache size 0 max 16384 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.533+0000 7f625c5cd740 1 Processor -- start 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.533+0000 7f625c5cd740 1 -- start start 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa357a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62541ae200 con 0x55a7fa34bfa0 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62541aba10 con 0x55a7fa340dc0 2024-04-25T18:57:31.535 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f625c5cd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62541a9880 con 0x55a7fa3416e0 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f6252000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f6252000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57894/0 (socket says 172.21.15.63:57894) 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f6253400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f6252000640 1 -- 172.21.15.63:0/3754830142 learned_addr learned my addr 172.21.15.63:0/3754830142 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.534+0000 7f6252a00640 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa357a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 -- 172.21.15.63:0/3754830142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 msgr2=0x55a7fa357a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa357a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.537 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 -- 172.21.15.63:0/3754830142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 msgr2=0x55a7fa34ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 -- 172.21.15.63:0/3754830142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fa347470 con 0x55a7fa3416e0 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252a00640 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa357a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f6252000640 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6240031440 tx=0x7f6240005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47c5c05b7099bd2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f623ea00640 1 -- 172.21.15.63:0/3754830142 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624004e040 con 0x55a7fa3416e0 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f623ea00640 1 -- 172.21.15.63:0/3754830142 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6240005520 con 0x55a7fa3416e0 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.535+0000 7f623ea00640 1 -- 172.21.15.63:0/3754830142 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240035040 con 0x55a7fa3416e0 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- 172.21.15.63:0/3754830142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 msgr2=0x55a7fa343780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6240031440 tx=0x7f6240005780 comp rx=0 tx=0).stop 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- 172.21.15.63:0/3754830142 shutdown_connections 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa34ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa3416e0 0x55a7fa343780 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 --2- 172.21.15.63:0/3754830142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa340dc0 0x55a7fa357a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- 172.21.15.63:0/3754830142 >> 172.21.15.63:0/3754830142 conn(0x55a7fa33d2d0 msgr2=0x55a7fa34a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- 172.21.15.63:0/3754830142 shutdown_connections 2024-04-25T18:57:31.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- 172.21.15.63:0/3754830142 wait complete. 2024-04-25T18:57:31.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.536+0000 7f625c5cd740 1 -- start start 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 Processor -- start 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa3416e0 0x55a7fa352070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa3525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62541a9ed0 con 0x55a7fa34bfa0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62541ab340 con 0x55a7fa3416e0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f62541aa5a0 con 0x55a7fa340dc0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f625c5cd740 10 client.0 _ll_register_callbacks cb 0x7f6240003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f6252a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f6252a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57908/0 (socket says 172.21.15.63:57908) 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f6252a00640 1 -- 172.21.15.63:0/4175341966 learned_addr learned my addr 172.21.15.63:0/4175341966 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f6253400640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa3525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.537+0000 7f6252000640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa3416e0 0x55a7fa352070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 -- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa3416e0 msgr2=0x55a7fa352070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa3416e0 0x55a7fa352070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 -- 172.21.15.63:0/4175341966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 msgr2=0x55a7fa3525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa3525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 -- 172.21.15.63:0/4175341966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fa3c1590 con 0x55a7fa340dc0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6253400640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa3525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6252a00640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55a7fa35eab0 tx=0x7f62440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d93b4a86253a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6233e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624402b500 con 0x55a7fa340dc0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6240005520 con 0x55a7fa340dc0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.538+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f624402b6a0 con 0x55a7fa340dc0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f624402b8a0 con 0x55a7fa340dc0 2024-04-25T18:57:31.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 10 client.26515 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a7fa3c28a0 con 0x55a7fa340dc0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6240005520 con 0x55a7fa340dc0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f624401f490 con 0x55a7fa340dc0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 10 client.26515 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 10 client.26515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 20 client.26515 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa3c2fc0 0x55a7fa3c5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 4 client.26515 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7fa2b0320 con 0x55a7fa3c2fc0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.539+0000 7f6252000640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa3c2fc0 0x55a7fa3c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.540+0000 7f6252000640 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa3c2fc0 0x55a7fa3c5400 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf0af28c7366541 server_cookie=253992a5b8afbdd2 in_seq=0 out_seq=0 2024-04-25T18:57:31.544 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.540+0000 7f6233e00640 10 client.26515 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f6233e00640 1 -- 172.21.15.63:0/4175341966 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7fa2b0320 con 0x55a7fa3c2fc0 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f6233e00640 10 client.26515 handle_command_reply: tid=0 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 client.26515 shutdown 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 msgr2=0x55a7fa351b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55a7fa35eab0 tx=0x7f62440046c0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 shutdown_connections 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a7fa3c2fc0 0x55a7fa3c5400 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a7fa3416e0 0x55a7fa352070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a7fa34bfa0 0x55a7fa3525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f6233e00640 0 client.26515 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 --2- 172.21.15.63:0/4175341966 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a7fa340dc0 0x55a7fa351b30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 >> 172.21.15.63:0/4175341966 conn(0x55a7fa33d2d0 msgr2=0x55a7fa349730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.575+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 shutdown_connections 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.576+0000 7f625c5cd740 1 -- 172.21.15.63:0/4175341966 wait complete. 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.576+0000 7f625c5cd740 20 client.26515 trim_cache size 0 max 16384 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:57:31.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 msgr2=0x7f62380a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 0x7f62380a61d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55a7fa342360 tx=0x7f6244005f90 comp rx=0 tx=0).stop 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 msgr2=0x7f625419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 0x7f625419ce70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f624c00da10 tx=0x7f624c00ded0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 shutdown_connections 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f62380a3d10 0x7f62380a61d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f625410c370 0x7f625419ce70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6254103870 0x7f6254198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 --2- 172.21.15.63:0/1533999179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6254102f50 0x7f62541985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 >> 172.21.15.63:0/1533999179 conn(0x7f62540fc770 msgr2=0x7f62540fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 shutdown_connections 2024-04-25T18:57:31.580 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:31.577+0000 7f6253e00640 1 -- 172.21.15.63:0/1533999179 wait complete. 2024-04-25T18:57:32.508 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.43 MB/sec execute 451 sec latency 1382.943 ms 2024-04-25T18:57:32.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:32 smithi114 ceph-mon[32988]: pgmap v344: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:32.600 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:32 smithi063 ceph-mon[28006]: pgmap v344: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:32 smithi070 ceph-mon[32820]: pgmap v344: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:33.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.42 MB/sec execute 452 sec latency 2383.047 ms 2024-04-25T18:57:34.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.41 MB/sec execute 453 sec latency 3383.217 ms 2024-04-25T18:57:34.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:34 smithi114 ceph-mon[32988]: pgmap v345: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:34 smithi063 ceph-mon[28006]: pgmap v345: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:34.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:34 smithi070 ceph-mon[32820]: pgmap v345: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 135 op/s 2024-04-25T18:57:35.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.41 MB/sec execute 454 sec latency 4383.400 ms 2024-04-25T18:57:36.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.40 MB/sec execute 455 sec latency 5383.495 ms 2024-04-25T18:57:36.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:36 smithi114 ceph-mon[32988]: pgmap v346: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T18:57:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:36 smithi063 ceph-mon[28006]: pgmap v346: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T18:57:36.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:36 smithi070 ceph-mon[32820]: pgmap v346: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 192 op/s 2024-04-25T18:57:37.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.39 MB/sec execute 456 sec latency 6383.606 ms 2024-04-25T18:57:38.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351754 3.38 MB/sec execute 457 sec latency 7383.749 ms 2024-04-25T18:57:38.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:38 smithi114 ceph-mon[32988]: pgmap v347: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T18:57:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:38 smithi063 ceph-mon[28006]: pgmap v347: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T18:57:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:38 smithi070 ceph-mon[32820]: pgmap v347: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 136 op/s 2024-04-25T18:57:39.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351758 3.38 MB/sec execute 458 sec latency 7907.396 ms 2024-04-25T18:57:40.509 INFO:tasks.workunit.client.0.smithi063.stdout: 1 351758 3.37 MB/sec execute 459 sec latency 1473.453 ms 2024-04-25T18:57:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:40 smithi063 ceph-mon[28006]: pgmap v348: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-25T18:57:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:40 smithi070 ceph-mon[32820]: pgmap v348: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-25T18:57:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:40 smithi114 ceph-mon[32988]: pgmap v348: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 154 op/s 2024-04-25T18:57:41.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 352408 3.37 MB/sec execute 460 sec latency 1860.250 ms 2024-04-25T18:57:42.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 353656 3.37 MB/sec execute 461 sec latency 23.823 ms 2024-04-25T18:57:42.618 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:42 smithi114 ceph-mon[32988]: pgmap v349: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:42 smithi063 ceph-mon[28006]: pgmap v349: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:42 smithi070 ceph-mon[32820]: pgmap v349: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:43.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 354928 3.39 MB/sec execute 462 sec latency 17.875 ms 2024-04-25T18:57:44.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 355422 3.38 MB/sec execute 463 sec latency 25.090 ms 2024-04-25T18:57:44.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:44 smithi063 ceph-mon[28006]: pgmap v350: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:44 smithi070 ceph-mon[32820]: pgmap v350: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 -- 172.21.15.63:0/2058315767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e81043b0 msgr2=0x7f08e8104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 --2- 172.21.15.63:0/2058315767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e81043b0 0x7f08e8104790 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009900 tx=0x7f08d802a3f0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 -- 172.21.15.63:0/2058315767 shutdown_connections 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 --2- 172.21.15.63:0/2058315767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8104d60 0x7f08e8108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 --2- 172.21.15.63:0/2058315767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e8109480 0x7f08e8111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 --2- 172.21.15.63:0/2058315767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e81043b0 0x7f08e8104790 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 -- 172.21.15.63:0/2058315767 >> 172.21.15.63:0/2058315767 conn(0x7f08e80ffc50 msgr2=0x7f08e8102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 -- 172.21.15.63:0/2058315767 shutdown_connections 2024-04-25T18:57:44.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.736+0000 7f08eea00640 1 -- 172.21.15.63:0/2058315767 wait complete. 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.737+0000 7f08eea00640 1 Processor -- start 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.737+0000 7f08eea00640 1 -- start start 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 0x7f08e81a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 0x7f08e819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08e8114430 con 0x7f08e81043b0 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81142b0 con 0x7f08e8109480 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eea00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81145b0 con 0x7f08e819a710 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 0x7f08e819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52034/0 (socket says 172.21.15.63:52034) 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08e7e00640 1 -- 172.21.15.63:0/78030915 learned_addr learned my addr 172.21.15.63:0/78030915 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.738+0000 7f08e7400640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 0x7f08e81a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 -- 172.21.15.63:0/78030915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 msgr2=0x7f08e81a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 0x7f08e81a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 -- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 msgr2=0x7f08e81a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 -- 172.21.15.63:0/78030915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e819b310 con 0x7f08e819a710 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08e7400640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 0x7f08e81a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08e7e00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eca00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 0x7f08e819ab90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009fd0 tx=0x7f08e000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08e4c00640 1 -- 172.21.15.63:0/78030915 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e000ce20 con 0x7f08e819a710 2024-04-25T18:57:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08e4c00640 1 -- 172.21.15.63:0/78030915 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08e0005020 con 0x7f08e819a710 2024-04-25T18:57:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08e4c00640 1 -- 172.21.15.63:0/78030915 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0010690 con 0x7f08e819a710 2024-04-25T18:57:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.739+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e81a73f0 con 0x7f08e819a710 2024-04-25T18:57:44.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.740+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e81a77f0 con 0x7f08e819a710 2024-04-25T18:57:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.743+0000 7f08e4c00640 1 -- 172.21.15.63:0/78030915 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f08e00054d0 con 0x7f08e819a710 2024-04-25T18:57:44.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.743+0000 7f08f17dd740 1 Processor -- start 2024-04-25T18:57:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.743+0000 7f08f17dd740 1 -- start start 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd421dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bd4183900 con 0x561bd421d820 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bd4129ef0 con 0x561bd421e140 2024-04-25T18:57:44.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08f17dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bd417f8c0 con 0x561bd4228a00 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36776/0 (socket says 172.21.15.63:36776) 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08e7e00640 1 -- 172.21.15.63:0/2145592042 learned_addr learned my addr 172.21.15.63:0/2145592042 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08e7400640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd421dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.744+0000 7f08e7400640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 -- 172.21.15.63:0/2145592042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 msgr2=0x561bd422adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 -- 172.21.15.63:0/2145592042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 msgr2=0x561bd421dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd421dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 -- 172.21.15.63:0/2145592042 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bd3e96210 con 0x561bd421e140 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7400640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:44.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e7e00640 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f08d8006010 tx=0x7f08d80044b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de3ba2f4c1bd932 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08e4c00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 0x7f08cc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08dd200640 1 -- 172.21.15.63:0/2145592042 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8040070 con 0x561bd421e140 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08dd200640 1 -- 172.21.15.63:0/2145592042 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d80047a0 con 0x561bd421e140 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.745+0000 7f08dd200640 1 -- 172.21.15.63:0/2145592042 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8035040 con 0x561bd421e140 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.746+0000 7f08e4c00640 1 -- 172.21.15.63:0/78030915 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f08e00d0e00 con 0x7f08e819a710 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.746+0000 7f08e7400640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 0x7f08cc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.746+0000 7f08f17dd740 1 -- 172.21.15.63:0/2145592042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 msgr2=0x561bd42284c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.746+0000 7f08f17dd740 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f08d8006010 tx=0x7f08d80044b0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08e7400640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 0x7f08cc0a6300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x561bd421b150 tx=0x7f08d4004730 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 -- 172.21.15.63:0/2145592042 shutdown_connections 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd42284c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 --2- 172.21.15.63:0/2145592042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd421dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 -- 172.21.15.63:0/2145592042 >> 172.21.15.63:0/2145592042 conn(0x561bd4219d30 msgr2=0x561bd421a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 -- 172.21.15.63:0/2145592042 shutdown_connections 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 -- 172.21.15.63:0/2145592042 wait complete. 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.747+0000 7f08f17dd740 1 -- start start 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 Processor -- start 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42258c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd4228a00 0x561bd4226340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561bd422b5d0 con 0x561bd421d820 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561bd413d310 con 0x561bd4228a00 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561bd422b750 con 0x561bd421e140 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08f17dd740 10 client.0 _ll_register_callbacks cb 0x561bd410bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08e7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08e7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55864/0 (socket says 172.21.15.63:55864) 2024-04-25T18:57:44.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.748+0000 7f08e7400640 1 -- 172.21.15.63:0/1944056841 learned_addr learned my addr 172.21.15.63:0/1944056841 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7e00640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42258c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08eca00640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd4228a00 0x561bd4226340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 -- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd4228a00 msgr2=0x561bd4226340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd4228a00 0x561bd4226340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 -- 172.21.15.63:0/1944056841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 msgr2=0x561bd42258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42258c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 -- 172.21.15.63:0/1944056841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bd3f62620 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08e7400640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f08d4013fd0 tx=0x7f08d4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6389bd0841b7376 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.749+0000 7f08d1600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d402f060 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f08d4020960 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d80047a0 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f08d4004180 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08f17dd740 10 client.26527 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.750+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561bd4237830 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d80047a0 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f08d402ab30 con 0x561bd421e140 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 10 client.26527 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:44.755 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 10 client.26527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 20 client.26527 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd4238db0 0x561bd423b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 4 client.26527 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bd41978b0 con 0x561bd4238db0 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.751+0000 7f08e7e00640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd4238db0 0x561bd423b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.752+0000 7f08e7e00640 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd4238db0 0x561bd423b1d0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed222b01080d763 server_cookie=9d7b45ae9a4d1eab in_seq=0 out_seq=0 2024-04-25T18:57:44.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.752+0000 7f08d1600640 10 client.26527 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08d1600640 1 -- 172.21.15.63:0/1944056841 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bd41978b0 con 0x561bd4238db0 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08d1600640 10 client.26527 handle_command_reply: tid=0 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 client.26527 shutdown 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 msgr2=0x561bd4225e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f08d4013fd0 tx=0x7f08d4005f90 comp rx=0 tx=0).stop 2024-04-25T18:57:44.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 shutdown_connections 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd4238db0 0x561bd423b1d0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd4228a00 0x561bd4226340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd421e140 0x561bd4225e00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 --2- 172.21.15.63:0/1944056841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42258c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 >> 172.21.15.63:0/1944056841 conn(0x561bd4219d30 msgr2=0x561bd421fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.753+0000 7f08d1600640 0 client.26527 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.754+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 shutdown_connections 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.754+0000 7f08f17dd740 1 -- 172.21.15.63:0/1944056841 wait complete. 2024-04-25T18:57:44.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.754+0000 7f08f17dd740 20 client.26527 trim_cache size 0 max 16384 2024-04-25T18:57:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.772+0000 7f08f17dd740 1 Processor -- start 2024-04-25T18:57:44.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.772+0000 7f08f17dd740 1 -- start start 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.772+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.772+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421e140 0x561bd4240ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.772+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08f17dd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81b2780 con 0x561bd421e140 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08f17dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81aff90 con 0x561bd421d820 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08f17dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81ade00 con 0x561bd4228a00 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.776 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421e140 0x561bd4240ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55876/0 (socket says 172.21.15.63:55876) 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08eca00640 1 -- 172.21.15.63:0/377165879 learned_addr learned my addr 172.21.15.63:0/377165879 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36804/0 (socket says 172.21.15.63:36804) 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 -- 172.21.15.63:0/377165879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 msgr2=0x561bd4241020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 -- 172.21.15.63:0/377165879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421e140 msgr2=0x561bd4240ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421e140 0x561bd4240ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08e7e00640 1 -- 172.21.15.63:0/377165879 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bd422b110 con 0x561bd421d820 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.773+0000 7f08eca00640 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.774+0000 7f08e7e00640 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f08d8005ec0 tx=0x7f08d8003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c37b8dd4bb63e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.774+0000 7f08d3e00640 1 -- 172.21.15.63:0/377165879 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8040070 con 0x561bd421d820 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.774+0000 7f08d3e00640 1 -- 172.21.15.63:0/377165879 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d80037a0 con 0x561bd421d820 2024-04-25T18:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.774+0000 7f08d3e00640 1 -- 172.21.15.63:0/377165879 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d8035040 con 0x561bd421d820 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- 172.21.15.63:0/377165879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 msgr2=0x561bd42344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f08d8005ec0 tx=0x7f08d8003730 comp rx=0 tx=0).stop 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- 172.21.15.63:0/377165879 shutdown_connections 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd4241020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421e140 0x561bd4240ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 --2- 172.21.15.63:0/377165879 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421d820 0x561bd42344c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- 172.21.15.63:0/377165879 >> 172.21.15.63:0/377165879 conn(0x561bd4219d30 msgr2=0x561bd4234ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- 172.21.15.63:0/377165879 shutdown_connections 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- 172.21.15.63:0/377165879 wait complete. 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.775+0000 7f08f17dd740 1 -- start start 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 Processor -- start 2024-04-25T18:57:44.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81ae450 con 0x561bd421d820 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81af8c0 con 0x561bd421e140 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f08e81aeb20 con 0x561bd4228a00 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08f17dd740 10 client.0 _ll_register_callbacks cb 0x7f08d40039f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08e7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.776+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55882/0 (socket says 172.21.15.63:55882) 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08e7400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:36818/0 (socket says 172.21.15.63:36818) 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 -- 172.21.15.63:0/3794952789 learned_addr learned my addr 172.21.15.63:0/3794952789 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 -- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 msgr2=0x561bd4230c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 -- 172.21.15.63:0/3794952789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 msgr2=0x561bd42306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42306c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 -- 172.21.15.63:0/3794952789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bd422e4a0 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08e7400640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08eca00640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f08e000c760 tx=0x7f08e00a1110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b613921aa90fa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08d1600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.777+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e00ba020 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f08e00993d0 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08d80037a0 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f08e00a5540 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 10 client.26533 fetch_fsmap learned FSMap version 43 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x561bd429eeb0 con 0x561bd4228a00 2024-04-25T18:57:44.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08d80037a0 con 0x561bd4228a00 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 43) ==== 8358+0+0 (secure 0 0 0) 0x7f08e0081030 con 0x561bd4228a00 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 10 client.26533 fetch_fsmap finished waiting for FSMap version 43 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 10 client.26533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 20 client.26533 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd429f650 0x561bd42a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 4 client.26533 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.778+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561bd4191320 con 0x561bd429f650 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.779+0000 7f08e7e00640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd429f650 0x561bd42a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:44.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.779+0000 7f08e7e00640 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd429f650 0x561bd42a1a70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de5bda0b47293dc server_cookie=c9e9f17514320a6c in_seq=0 out_seq=0 2024-04-25T18:57:44.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.779+0000 7f08d1600640 10 client.26533 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08d1600640 1 -- 172.21.15.63:0/3794952789 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561bd4191320 con 0x561bd429f650 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08d1600640 10 client.26533 handle_command_reply: tid=0 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 client.26533 shutdown 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 msgr2=0x561bd422ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f08e000c760 tx=0x7f08e00a1110 comp rx=0 tx=0).stop 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 shutdown_connections 2024-04-25T18:57:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561bd429f650 0x561bd42a1a70 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561bd421e140 0x561bd4230c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.810+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561bd4228a00 0x561bd422ddb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08d1600640 0 client.26533 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08f17dd740 1 --2- 172.21.15.63:0/3794952789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561bd421d820 0x561bd42306c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 >> 172.21.15.63:0/3794952789 conn(0x561bd4219d30 msgr2=0x561bd421f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 shutdown_connections 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08f17dd740 1 -- 172.21.15.63:0/3794952789 wait complete. 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.811+0000 7f08f17dd740 20 client.26533 trim_cache size 0 max 16384 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:57:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 msgr2=0x7f08cc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 0x7f08cc0a6300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x561bd421b150 tx=0x7f08d4004730 comp rx=0 tx=0).stop 2024-04-25T18:57:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 msgr2=0x7f08e819ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 0x7f08e819ab90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009fd0 tx=0x7f08e000ef10 comp rx=0 tx=0).stop 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 shutdown_connections 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f08cc0a3e40 0x7f08cc0a6300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f08e8109480 0x7f08e81a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f08e819a710 0x7f08e819ab90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 --2- 172.21.15.63:0/78030915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08e81043b0 0x7f08e81a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 >> 172.21.15.63:0/78030915 conn(0x7f08e80ffc50 msgr2=0x7f08e8101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 shutdown_connections 2024-04-25T18:57:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:44.812+0000 7f08eea00640 1 -- 172.21.15.63:0/78030915 wait complete. 2024-04-25T18:57:44.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:44 smithi114 ceph-mon[32988]: pgmap v350: 97 pgs: 97 active+clean; 107 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 814 KiB/s wr, 74 op/s 2024-04-25T18:57:45.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 356349 3.39 MB/sec execute 464 sec latency 55.522 ms 2024-04-25T18:57:46.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 357451 3.39 MB/sec execute 465 sec latency 59.019 ms 2024-04-25T18:57:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:46 smithi063 ceph-mon[28006]: pgmap v351: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T18:57:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:46 smithi070 ceph-mon[32820]: pgmap v351: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T18:57:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:46 smithi114 ceph-mon[32988]: pgmap v351: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 1.3 MiB/s wr, 125 op/s 2024-04-25T18:57:47.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 358490 3.40 MB/sec execute 466 sec latency 18.823 ms 2024-04-25T18:57:48.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 358982 3.40 MB/sec execute 467 sec latency 25.737 ms 2024-04-25T18:57:48.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:48 smithi063 ceph-mon[28006]: pgmap v352: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 838 KiB/s wr, 68 op/s 2024-04-25T18:57:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:48 smithi070 ceph-mon[32820]: pgmap v352: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 838 KiB/s wr, 68 op/s 2024-04-25T18:57:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:48 smithi114 ceph-mon[32988]: pgmap v352: 97 pgs: 97 active+clean; 110 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 838 KiB/s wr, 68 op/s 2024-04-25T18:57:49.510 INFO:tasks.workunit.client.0.smithi063.stdout: 1 359908 3.40 MB/sec execute 468 sec latency 35.214 ms 2024-04-25T18:57:50.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 359941 3.39 MB/sec execute 469 sec latency 959.899 ms 2024-04-25T18:57:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:50 smithi063 ceph-mon[28006]: pgmap v353: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:57:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:50 smithi070 ceph-mon[32820]: pgmap v353: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:57:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:50 smithi114 ceph-mon[32988]: pgmap v353: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-04-25T18:57:51.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 360735 3.39 MB/sec execute 470 sec latency 1355.973 ms 2024-04-25T18:57:52.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 361903 3.40 MB/sec execute 471 sec latency 20.120 ms 2024-04-25T18:57:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:52 smithi063 ceph-mon[28006]: pgmap v354: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1007 KiB/s wr, 120 op/s 2024-04-25T18:57:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:52 smithi070 ceph-mon[32820]: pgmap v354: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1007 KiB/s wr, 120 op/s 2024-04-25T18:57:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:52 smithi114 ceph-mon[32988]: pgmap v354: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1007 KiB/s wr, 120 op/s 2024-04-25T18:57:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:57:53.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 362352 3.40 MB/sec execute 472 sec latency 20.246 ms 2024-04-25T18:57:54.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 363310 3.40 MB/sec execute 473 sec latency 27.760 ms 2024-04-25T18:57:54.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:54 smithi063 ceph-mon[28006]: pgmap v355: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 120 op/s 2024-04-25T18:57:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:54 smithi070 ceph-mon[32820]: pgmap v355: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 120 op/s 2024-04-25T18:57:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:54 smithi114 ceph-mon[32988]: pgmap v355: 97 pgs: 97 active+clean; 111 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 120 op/s 2024-04-25T18:57:55.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 363483 3.40 MB/sec execute 474 sec latency 810.644 ms 2024-04-25T18:57:56.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 364093 3.39 MB/sec execute 475 sec latency 1496.977 ms 2024-04-25T18:57:56.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:56 smithi063 ceph-mon[28006]: pgmap v356: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-04-25T18:57:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:56 smithi070 ceph-mon[32820]: pgmap v356: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-04-25T18:57:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:56 smithi114 ceph-mon[32988]: pgmap v356: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-04-25T18:57:57.511 INFO:tasks.workunit.client.0.smithi063.stdout: 1 365155 3.40 MB/sec execute 476 sec latency 18.745 ms 2024-04-25T18:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 -- 172.21.15.63:0/1663121520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0102980 msgr2=0x7f6ab01069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/1663121520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0102980 0x7f6ab01069c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009860 tx=0x7f6a9c02a340 comp rx=0 tx=0).stop 2024-04-25T18:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 -- 172.21.15.63:0/1663121520 shutdown_connections 2024-04-25T18:57:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/1663121520 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0102980 0x7f6ab01069c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/1663121520 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/1663121520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 -- 172.21.15.63:0/1663121520 >> 172.21.15.63:0/1663121520 conn(0x7f6ab00fd8d0 msgr2=0x7f6ab00ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 -- 172.21.15.63:0/1663121520 shutdown_connections 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.959+0000 7f6ab6a00640 1 -- 172.21.15.63:0/1663121520 wait complete. 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.960+0000 7f6ab6a00640 1 Processor -- start 2024-04-25T18:57:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.960+0000 7f6ab6a00640 1 -- start start 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab019ce80 0x7f6ab019d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab0111fc0 con 0x7f6ab019ce80 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab0111e40 con 0x7f6ab0101fd0 2024-04-25T18:57:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6a00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab0112140 con 0x7f6ab01070a0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab019ce80 0x7f6ab019d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab5600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56790/0 (socket says 172.21.15.63:56790) 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.961+0000 7f6ab4c00640 1 -- 172.21.15.63:0/499046036 learned_addr learned my addr 172.21.15.63:0/499046036 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 -- 172.21.15.63:0/499046036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 msgr2=0x7f6ab0198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 -- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab019ce80 msgr2=0x7f6ab019d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab019ce80 0x7f6ab019d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 -- 172.21.15.63:0/499046036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab019d970 con 0x7f6ab0101fd0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab5600640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01985c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4005fd0 tx=0x7f6aa400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6aae000640 1 -- 172.21.15.63:0/499046036 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400ccc0 con 0x7f6ab0101fd0 2024-04-25T18:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6aae000640 1 -- 172.21.15.63:0/499046036 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aa40035f0 con 0x7f6ab0101fd0 2024-04-25T18:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab019dc60 con 0x7f6ab0101fd0 2024-04-25T18:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6aae000640 1 -- 172.21.15.63:0/499046036 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400ce60 con 0x7f6ab0101fd0 2024-04-25T18:57:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.962+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab01a54a0 con 0x7f6ab0101fd0 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.967+0000 7f6abaddd740 1 Processor -- start 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.967+0000 7f6abaddd740 1 -- start start 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aeee650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6aae000640 1 -- 172.21.15.63:0/499046036 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6aa4003aa0 con 0x7f6ab0101fd0 2024-04-25T18:57:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562e5ae49900 con 0x562e5aeeeb90 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e5adefef0 con 0x562e5aee42d0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6abaddd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562e5ae458c0 con 0x562e5aee39b0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56800/0 (socket says 172.21.15.63:56800) 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 -- 172.21.15.63:0/2978033119 learned_addr learned my addr 172.21.15.63:0/2978033119 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab5600640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aeee650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 -- 172.21.15.63:0/2978033119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 msgr2=0x562e5aeee650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aeee650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 -- 172.21.15.63:0/2978033119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 msgr2=0x562e5aef0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab6000640 1 -- 172.21.15.63:0/2978033119 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e5ab5c210 con 0x562e5aee39b0 2024-04-25T18:57:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.968+0000 7f6ab5600640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aeee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.969+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009f60 tx=0x7f6aa8009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1368fb0c54d04cad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.969+0000 7f6aa2600640 1 -- 172.21.15.63:0/2978033119 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa801e070 con 0x562e5aee39b0 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.969+0000 7f6aa2600640 1 -- 172.21.15.63:0/2978033119 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aa8003c30 con 0x562e5aee39b0 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.969+0000 7f6aa2600640 1 -- 172.21.15.63:0/2978033119 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa800f8a0 con 0x562e5aee39b0 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6aae000640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 0x7f6a980a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 -- 172.21.15.63:0/2978033119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 msgr2=0x562e5aee3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009f60 tx=0x7f6aa8009f90 comp rx=0 tx=0).stop 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 -- 172.21.15.63:0/2978033119 shutdown_connections 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aeee650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2978033119 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aee3d90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 -- 172.21.15.63:0/2978033119 >> 172.21.15.63:0/2978033119 conn(0x562e5aedfec0 msgr2=0x562e5aee02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6aae000640 1 -- 172.21.15.63:0/499046036 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f6aa40d0a30 con 0x7f6ab0101fd0 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6ab5600640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 0x7f6a980a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 -- 172.21.15.63:0/2978033119 shutdown_connections 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.970+0000 7f6abaddd740 1 -- 172.21.15.63:0/2978033119 wait complete. 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.971+0000 7f6abaddd740 1 -- start start 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.971+0000 7f6ab5600640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 0x7f6a980a6250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40ce1f0 tx=0x7f6aa4003a10 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.971+0000 7f6abaddd740 1 Processor -- start 2024-04-25T18:57:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aeebb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aeec5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562e5aef17c0 con 0x562e5aeeeb90 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562e5ae03310 con 0x562e5aee39b0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562e5aef1940 con 0x562e5aee42d0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6abaddd740 10 client.0 _ll_register_callbacks cb 0x562e5add1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aeebb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56802/0 (socket says 172.21.15.63:56802) 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2410891338 learned_addr learned my addr 172.21.15.63:0/2410891338 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aeec5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 msgr2=0x562e5aeebb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aeebb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2410891338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 msgr2=0x562e5aeec5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aeec5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2410891338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e5ac28620 con 0x562e5aee42d0 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.972+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aeec5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c000c00 tx=0x7f6a9c009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=357bb0aec12b59bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6a96a00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c037040 con 0x562e5aee42d0 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f6a9c030450 con 0x562e5aee42d0 2024-04-25T18:57:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6aa8003c30 con 0x562e5aee42d0 2024-04-25T18:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6a9c04c050 con 0x562e5aee42d0 2024-04-25T18:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6abaddd740 10 client.26545 fetch_fsmap learned FSMap version 44 2024-04-25T18:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.973+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562e5af01e80 con 0x562e5aee42d0 2024-04-25T18:57:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aa8003c30 con 0x562e5aee42d0 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f6a9c0035f0 con 0x562e5aee42d0 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 10 client.26545 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 10 client.26545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 20 client.26545 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af03400 0x562e5af05810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 4 client.26545 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:57:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e5ae5d8b0 con 0x562e5af03400 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.974+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af03400 0x562e5af05810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.975+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af03400 0x562e5af05810 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70cacfe46303c17e server_cookie=cc52da9ff5d67453 in_seq=0 out_seq=0 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.975+0000 7f6a96a00640 10 client.26545 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.975+0000 7f6a96a00640 1 -- 172.21.15.63:0/2410891338 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e5ae5d8b0 con 0x562e5af03400 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.975+0000 7f6a96a00640 10 client.26545 handle_command_reply: tid=0 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 client.26545 shutdown 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 msgr2=0x562e5aeec0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c000c00 tx=0x7f6a9c009490 comp rx=0 tx=0).stop 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 shutdown_connections 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af03400 0x562e5af05810 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aeec5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee42d0 0x562e5aeec0b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2410891338 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aeebb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6a96a00640 0 client.26545 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 >> 172.21.15.63:0/2410891338 conn(0x562e5aedfec0 msgr2=0x562e5aee5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 shutdown_connections 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 1 -- 172.21.15.63:0/2410891338 wait complete. 2024-04-25T18:57:57.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.976+0000 7f6abaddd740 20 client.26545 trim_cache size 0 max 16384 2024-04-25T18:57:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 Processor -- start 2024-04-25T18:57:57.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 -- start start 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aefead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aeeeb90 0x562e5aedeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01ae380 con 0x562e5aee42d0 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01abb90 con 0x562e5aee39b0 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.993+0000 7f6abaddd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01a99e0 con 0x562e5aeeeb90 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37778/0 (socket says 172.21.15.63:37778) 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2013196138 learned_addr learned my addr 172.21.15.63:0/2013196138 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aefead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aeeeb90 0x562e5aedeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2013196138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aeeeb90 msgr2=0x562e5aedeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aeeeb90 0x562e5aedeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2013196138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 msgr2=0x562e5aefead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aefead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2013196138 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e5aeea0c0 con 0x562e5aee42d0 2024-04-25T18:57:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.994+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aefead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:57:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.995+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009250 tx=0x7f6a9c02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedd55c70ee5127 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.995+0000 7f6aa1c00640 1 -- 172.21.15.63:0/2013196138 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c037040 con 0x562e5aee42d0 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.995+0000 7f6aa1c00640 1 -- 172.21.15.63:0/2013196138 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a9c030900 con 0x562e5aee42d0 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.995+0000 7f6aa1c00640 1 -- 172.21.15.63:0/2013196138 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0327f0 con 0x562e5aee42d0 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- 172.21.15.63:0/2013196138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 msgr2=0x562e5aeec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009250 tx=0x7f6a9c02a340 comp rx=0 tx=0).stop 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- 172.21.15.63:0/2013196138 shutdown_connections 2024-04-25T18:57:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aeeeb90 0x562e5aedeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aee42d0 0x562e5aeec410 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2013196138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee39b0 0x562e5aefead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- 172.21.15.63:0/2013196138 >> 172.21.15.63:0/2013196138 conn(0x562e5aedfec0 msgr2=0x562e5aede790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- 172.21.15.63:0/2013196138 shutdown_connections 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- 172.21.15.63:0/2013196138 wait complete. 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.996+0000 7f6abaddd740 1 -- start start 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 Processor -- start 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aefb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01aa050 con 0x562e5aeeeb90 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01ab4c0 con 0x562e5aee42d0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab01aa720 con 0x562e5aee39b0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6abaddd740 10 client.0 _ll_register_callbacks cb 0x7f6a9c0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6ab4c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56994/0 (socket says 172.21.15.63:56994) 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.997+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2493672493 learned_addr learned my addr 172.21.15.63:0/2493672493 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2493672493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 msgr2=0x562e5aefb420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aefb420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2493672493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 msgr2=0x562e5aef8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 -- 172.21.15.63:0/2493672493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e5aef9540 con 0x562e5aee42d0 2024-04-25T18:57:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6ab4c00640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0057c0 tx=0x7f6a9c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91884e7951913c72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6a96a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c05e060 con 0x562e5aee42d0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a9c030900 con 0x562e5aee42d0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.998+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 8587+0+0 (secure 0 0 0) 0x7f6a9c030450 con 0x562e5aee42d0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6a9c037040 con 0x562e5aee42d0 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 10 client.26531 fetch_fsmap learned FSMap version 44 2024-04-25T18:57:58.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562e5af69a40 con 0x562e5aee42d0 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a9c030900 con 0x562e5aee42d0 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f6a9c03d490 con 0x562e5aee42d0 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 10 client.26531 fetch_fsmap finished waiting for FSMap version 44 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 10 client.26531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 20 client.26531 populate_metadata read hostname 'smithi063' 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af6a080 0x562e5af6c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 4 client.26531 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:57:58.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e5ae57320 con 0x562e5af6a080 2024-04-25T18:57:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:57.999+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af6a080 0x562e5af6c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:57:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.000+0000 7f6ab6000640 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af6a080 0x562e5af6c4a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c02360caa4c4ef9c server_cookie=637888356e8c2c98 in_seq=0 out_seq=0 2024-04-25T18:57:58.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.000+0000 7f6a96a00640 10 client.26531 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.036+0000 7f6a96a00640 1 -- 172.21.15.63:0/2493672493 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e5ae57320 con 0x562e5af6a080 2024-04-25T18:57:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.036+0000 7f6a96a00640 10 client.26531 handle_command_reply: tid=0 2024-04-25T18:57:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.036+0000 7f6abaddd740 1 client.26531 shutdown 2024-04-25T18:57:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.036+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 msgr2=0x562e5aef87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.036+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0057c0 tx=0x7f6a9c005bd0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 shutdown_connections 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562e5af6a080 0x562e5af6c4a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562e5aee42d0 0x562e5aef87b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5aeeeb90 0x562e5aef8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 --2- 172.21.15.63:0/2493672493 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562e5aee39b0 0x562e5aefb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6a96a00640 0 client.26531 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 >> 172.21.15.63:0/2493672493 conn(0x562e5aedfec0 msgr2=0x562e5aee5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 shutdown_connections 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 1 -- 172.21.15.63:0/2493672493 wait complete. 2024-04-25T18:57:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.037+0000 7f6abaddd740 20 client.26531 trim_cache size 0 max 16384 2024-04-25T18:57:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:57:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:57:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:57:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:57:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 msgr2=0x7f6a980a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 0x7f6a980a6250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40ce1f0 tx=0x7f6aa4003a10 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 msgr2=0x7f6ab01985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01985c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4005fd0 tx=0x7f6aa400ed90 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 shutdown_connections 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f6a980a3d90 0x7f6a980a6250 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab019ce80 0x7f6ab019d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f6ab01070a0 0x7f6ab0198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 --2- 172.21.15.63:0/499046036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6ab0101fd0 0x7f6ab01985c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 >> 172.21.15.63:0/499046036 conn(0x7f6ab00fd8d0 msgr2=0x7f6ab00fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 shutdown_connections 2024-04-25T18:57:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:57:58.038+0000 7f6ab6a00640 1 -- 172.21.15.63:0/499046036 wait complete. 2024-04-25T18:57:58.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 365843 3.40 MB/sec execute 477 sec latency 22.132 ms 2024-04-25T18:57:58.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:57:58 smithi063 ceph-mon[28006]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 772 KiB/s wr, 115 op/s 2024-04-25T18:57:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:57:58 smithi070 ceph-mon[32820]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 772 KiB/s wr, 115 op/s 2024-04-25T18:57:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:57:58 smithi114 ceph-mon[32988]: pgmap v357: 97 pgs: 97 active+clean; 114 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 772 KiB/s wr, 115 op/s 2024-04-25T18:57:59.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 366793 3.41 MB/sec execute 478 sec latency 21.934 ms 2024-04-25T18:58:00.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 367032 3.40 MB/sec execute 479 sec latency 529.894 ms 2024-04-25T18:58:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:00 smithi063 ceph-mon[28006]: pgmap v358: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-04-25T18:58:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:00 smithi070 ceph-mon[32820]: pgmap v358: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-04-25T18:58:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:00 smithi114 ceph-mon[32988]: pgmap v358: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-04-25T18:58:01.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 367613 3.39 MB/sec execute 480 sec latency 913.636 ms 2024-04-25T18:58:02.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 368268 3.40 MB/sec execute 481 sec latency 54.509 ms 2024-04-25T18:58:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:02 smithi070 ceph-mon[32820]: pgmap v359: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:02 smithi114 ceph-mon[32988]: pgmap v359: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:02 smithi063 ceph-mon[28006]: pgmap v359: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:03.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 369265 3.41 MB/sec execute 482 sec latency 17.079 ms 2024-04-25T18:58:03.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:03 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:03 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:03.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:03 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:58:04.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 369956 3.40 MB/sec execute 483 sec latency 28.848 ms 2024-04-25T18:58:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:04 smithi070 ceph-mon[32820]: pgmap v360: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:04.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:04 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:04.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:04 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:04.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:04 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:58:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:04 smithi114 ceph-mon[32988]: pgmap v360: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:04 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:04 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:04 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:58:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:04 smithi063 ceph-mon[28006]: pgmap v360: 97 pgs: 97 active+clean; 112 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 992 KiB/s wr, 120 op/s 2024-04-25T18:58:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:04 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:58:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:04 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:58:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:04 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:58:04.949 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.0) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-04-25T18:58:04.949 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.5 2024-04-25T18:58:04.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:58:05.446 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:05.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 370575 3.40 MB/sec execute 484 sec latency 300.341 ms 2024-04-25T18:58:05.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:05 smithi070 ceph-mon[32820]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T18:58:05.791 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:05 smithi063 ceph-mon[28006]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T18:58:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:05 smithi114 ceph-mon[32988]: osdmap e110: 12 total, 12 up, 12 in 2024-04-25T18:58:06.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 371392 3.40 MB/sec execute 485 sec latency 681.984 ms 2024-04-25T18:58:06.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:06 smithi063 ceph-mon[28006]: pgmap v362: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:06 smithi070 ceph-mon[32820]: pgmap v362: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:06 smithi114 ceph-mon[32988]: pgmap v362: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:07.512 INFO:tasks.workunit.client.0.smithi063.stdout: 1 372561 3.42 MB/sec execute 486 sec latency 25.697 ms 2024-04-25T18:58:08.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.254+0000 7f8190600700 1 -- 172.21.15.63:0/1296203092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101360 msgr2=0x7f8188101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.254+0000 7f8190600700 1 --2- 172.21.15.63:0/1296203092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101360 0x7f8188101740 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f817c009f90 tx=0x7f817c00b330 comp rx=0 tx=0).stop 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 -- 172.21.15.63:0/1296203092 shutdown_connections 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 --2- 172.21.15.63:0/1296203092 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f81881063f0 0x7f818810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 --2- 172.21.15.63:0/1296203092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101360 0x7f8188101740 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 --2- 172.21.15.63:0/1296203092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8188101d10 0x7f8188105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 -- 172.21.15.63:0/1296203092 >> 172.21.15.63:0/1296203092 conn(0x7f8188077fb0 msgr2=0x7f81880ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 -- 172.21.15.63:0/1296203092 shutdown_connections 2024-04-25T18:58:08.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.255+0000 7f8190600700 1 -- 172.21.15.63:0/1296203092 wait complete. 2024-04-25T18:58:08.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.256+0000 7f8190600700 1 Processor -- start 2024-04-25T18:58:08.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.256+0000 7f8190600700 1 -- start start 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8188101360 0x7f8188199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 0x7f818819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8188110a50 con 0x7f81881063f0 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81881108d0 con 0x7f8188101d10 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f8190600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8188110bd0 con 0x7f8188101360 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f818d200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f818d200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60354/0 (socket says 172.21.15.63:60354) 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f818d200700 1 -- 172.21.15.63:0/3113094632 learned_addr learned my addr 172.21.15.63:0/3113094632 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.257+0000 7f818dc00700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8188101360 0x7f8188199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:08.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 -- 172.21.15.63:0/3113094632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8188101360 msgr2=0x7f8188199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:08.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818e600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 0x7f818819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:08.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8188101360 0x7f8188199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 -- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 msgr2=0x7f818819da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:08.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 0x7f818819da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 -- 172.21.15.63:0/3113094632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818819e170 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818e600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 0x7f818819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f818d200700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f817400e970 tx=0x7f817400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.258+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817400cbd0 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.259+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f817400cd70 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.259+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818819e460 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.259+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8174005680 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.259+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81881a5ce0 con 0x7f8188101d10 2024-04-25T18:58:08.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.260+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81881027f0 con 0x7f8188101d10 2024-04-25T18:58:08.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.262+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8174004750 con 0x7f8188101d10 2024-04-25T18:58:08.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.264+0000 7f8186000700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 0x7f81700a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:08.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.264+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7f81740d0a80 con 0x7f8188101d10 2024-04-25T18:58:08.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.264+0000 7f818dc00700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 0x7f81700a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:08.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.265+0000 7f818dc00700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 0x7f81700a62f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b6c0 tx=0x7f817c0096c0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:08.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.268+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f817409d0b0 con 0x7f8188101d10 2024-04-25T18:58:08.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 373098 3.41 MB/sec execute 487 sec latency 21.717 ms 2024-04-25T18:58:08.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.578+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f818802df80 con 0x7f8188101d10 2024-04-25T18:58:08.581 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:08 smithi063 ceph-mon[28006]: pgmap v363: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:08.582 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:58:08.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.579+0000 7f8186000700 1 -- 172.21.15.63:0/3113094632 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11877 (secure 0 0 0) 0x7f817409c9b0 con 0x7f8188101d10 2024-04-25T18:58:08.583 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:57:50.875058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:58:08.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.583+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 msgr2=0x7f81700a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:08.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.583+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 0x7f81700a62f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f817c00b6c0 tx=0x7f817c0096c0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.583+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 msgr2=0x7f81881996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:08.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.583+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f817400e970 tx=0x7f817400ec80 comp rx=0 tx=0).stop 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 shutdown_connections 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f81700a3e30 0x7f81700a62f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8188101360 0x7f8188199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81881063f0 0x7f818819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 --2- 172.21.15.63:0/3113094632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8188101d10 0x7f81881996a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 >> 172.21.15.63:0/3113094632 conn(0x7f8188077fb0 msgr2=0x7f81880ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 shutdown_connections 2024-04-25T18:58:08.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:08.584+0000 7f8190600700 1 -- 172.21.15.63:0/3113094632 wait complete. 2024-04-25T18:58:08.588 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 44 2024-04-25T18:58:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:08 smithi070 ceph-mon[32820]: pgmap v363: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:08.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:08 smithi114 ceph-mon[32988]: pgmap v363: 97 pgs: 97 active+clean; 115 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-25T18:58:09.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 373999 3.42 MB/sec execute 488 sec latency 29.238 ms 2024-04-25T18:58:09.579 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 81.5 --await-for 42.0' 2024-04-25T18:58:09.579 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 81.5 --await-for 42.0 2024-04-25T18:58:09.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:09 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3113094632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:09.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:09 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3113094632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:09.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:09 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3113094632' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:10.042 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:10.442 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:10 smithi063 ceph-mon[28006]: pgmap v364: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:10.443 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:10 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:10.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 374123 3.41 MB/sec execute 489 sec latency 778.430 ms 2024-04-25T18:58:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:10 smithi070 ceph-mon[32820]: pgmap v364: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:10 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:10 smithi114 ceph-mon[32988]: pgmap v364: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:10 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 -- 172.21.15.63:0/148000681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 msgr2=0x7f4dcc1126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/148000681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1126e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4db4009cd0 tx=0x7f4db402a8f0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 -- 172.21.15.63:0/148000681 shutdown_connections 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/148000681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 0x7f4dcc115010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/148000681 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1126e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/148000681 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 0x7f4dcc076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 -- 172.21.15.63:0/148000681 >> 172.21.15.63:0/148000681 conn(0x7f4dcc0ffa30 msgr2=0x7f4dcc101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 -- 172.21.15.63:0/148000681 shutdown_connections 2024-04-25T18:58:11.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.200+0000 7f4dcbe00640 1 -- 172.21.15.63:0/148000681 wait complete. 2024-04-25T18:58:11.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 Processor -- start 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 -- start start 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 0x7f4dcc1a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 0x7f4dcc19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc118630 con 0x7f4dcc076480 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1184b0 con 0x7f4dcc112c20 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcbe00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1187b0 con 0x7f4dcc076da0 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcaa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 0x7f4dcc1a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47568/0 (socket says 172.21.15.63:47568) 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 0x7f4dcc19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.202+0000 7f4dc2000640 1 -- 172.21.15.63:0/1018041952 learned_addr learned my addr 172.21.15.63:0/1018041952 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 -- 172.21.15.63:0/1018041952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 msgr2=0x7f4dcc19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 0x7f4dcc19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 -- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 msgr2=0x7f4dcc1a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 0x7f4dcc1a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 -- 172.21.15.63:0/1018041952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dcc19f1f0 con 0x7f4dcc076da0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc2000640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4db4006010 tx=0x7f4db4030cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc3e00640 1 -- 172.21.15.63:0/1018041952 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4004040 con 0x7f4dcc076da0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc3e00640 1 -- 172.21.15.63:0/1018041952 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4db4040070 con 0x7f4dcc076da0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dc3e00640 1 -- 172.21.15.63:0/1018041952 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4045b10 con 0x7f4dcc076da0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.203+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dcc19f480 con 0x7f4dcc076da0 2024-04-25T18:58:11.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.204+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dcc1ab730 con 0x7f4dcc076da0 2024-04-25T18:58:11.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.207+0000 7f4dc3e00640 1 -- 172.21.15.63:0/1018041952 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4db4045cb0 con 0x7f4dcc076da0 2024-04-25T18:58:11.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.207+0000 7f4dd464e740 1 Processor -- start 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.207+0000 7f4dd464e740 1 -- start start 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.207+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dd464e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56146a597900 con 0x56146a6322d0 2024-04-25T18:58:11.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dd464e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56146a53def0 con 0x56146a6319b0 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dd464e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56146a5938c0 con 0x56146a63cb90 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60372/0 (socket says 172.21.15.63:60372) 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 -- 172.21.15.63:0/3995148552 learned_addr learned my addr 172.21.15.63:0/3995148552 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dc2000640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 -- 172.21.15.63:0/3995148552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 msgr2=0x56146a63ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.208+0000 7f4dcb400640 1 -- 172.21.15.63:0/3995148552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 msgr2=0x56146a63c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dcb400640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dcb400640 1 -- 172.21.15.63:0/3995148552 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56146a2aa210 con 0x56146a6319b0 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dc2000640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:11.212 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dc3e00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 0x7f4db00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dcb400640 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009ba0 tx=0x7f4dc4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7bdbdd975029b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dc3e00640 1 -- 172.21.15.63:0/1018041952 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7f4db40f17a0 con 0x7f4dcc076da0 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 0x7f4db00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.209+0000 7f4dbbe00640 1 -- 172.21.15.63:0/3995148552 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4004230 con 0x56146a6319b0 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.210+0000 7f4dbbe00640 1 -- 172.21.15.63:0/3995148552 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dc4009e10 con 0x56146a6319b0 2024-04-25T18:58:11.213 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.210+0000 7f4dbbe00640 1 -- 172.21.15.63:0/3995148552 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4004d00 con 0x56146a6319b0 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- 172.21.15.63:0/3995148552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 msgr2=0x56146a631d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009ba0 tx=0x7f4dc4011810 comp rx=0 tx=0).stop 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 0x7f4db00a61d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x56146a632f50 tx=0x7f4dbc005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- 172.21.15.63:0/3995148552 shutdown_connections 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 --2- 172.21.15.63:0/3995148552 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a631d90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.215 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- 172.21.15.63:0/3995148552 >> 172.21.15.63:0/3995148552 conn(0x56146a62dec0 msgr2=0x56146a62e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- 172.21.15.63:0/3995148552 shutdown_connections 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- 172.21.15.63:0/3995148552 wait complete. 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.211+0000 7f4dd464e740 1 -- start start 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 Processor -- start 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56146a63f7c0 con 0x56146a6322d0 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56146a551310 con 0x56146a6319b0 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56146a63f940 con 0x56146a63cb90 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dcaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dd464e740 10 client.0 _ll_register_callbacks cb 0x56146a51fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dcaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60384/0 (socket says 172.21.15.63:60384) 2024-04-25T18:58:11.216 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.212+0000 7f4dcaa00640 1 -- 172.21.15.63:0/515027747 learned_addr learned my addr 172.21.15.63:0/515027747 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcb400640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dc2000640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 -- 172.21.15.63:0/515027747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 msgr2=0x56146a63a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 -- 172.21.15.63:0/515027747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 msgr2=0x56146a63a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 -- 172.21.15.63:0/515027747 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56146a376620 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0045f0 tx=0x7f4dbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c6da16a1183d79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.213+0000 7f4db8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc01f640 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7f4dbc01f7e0 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc4009e10 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f4dbc01f9e0 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4dd464e740 10 client.26549 fetch_fsmap learned FSMap version 45 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56146a64baa0 con 0x56146a6319b0 2024-04-25T18:58:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.214+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dc4009e10 con 0x56146a6319b0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f4dbc027070 con 0x56146a6319b0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 10 client.26549 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 10 client.26549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 20 client.26549 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a64cfb0 0x56146a64f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 4 client.26549 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56146a5ab8b0 con 0x56146a64cfb0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.215+0000 7f4dc2000640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a64cfb0 0x56146a64f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.216+0000 7f4db8c00640 10 client.26549 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.216+0000 7f4dc2000640 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a64cfb0 0x56146a64f420 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a864b392350ea74 server_cookie=478663d871f1f60c in_seq=0 out_seq=0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.216+0000 7f4db8c00640 1 -- 172.21.15.63:0/515027747 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56146a5ab8b0 con 0x56146a64cfb0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.216+0000 7f4db8c00640 10 client.26549 handle_command_reply: tid=0 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.216+0000 7f4dd464e740 1 client.26549 shutdown 2024-04-25T18:58:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 msgr2=0x56146a639b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0045f0 tx=0x7f4dbc004730 comp rx=0 tx=0).stop 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 shutdown_connections 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a64cfb0 0x56146a64f420 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a63a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6322d0 0x56146a63a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4db8c00640 0 client.26549 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 --2- 172.21.15.63:0/515027747 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6319b0 0x56146a639b40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 >> 172.21.15.63:0/515027747 conn(0x56146a62dec0 msgr2=0x56146a633c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 shutdown_connections 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 1 -- 172.21.15.63:0/515027747 wait complete. 2024-04-25T18:58:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.217+0000 7f4dd464e740 20 client.26549 trim_cache size 0 max 16384 2024-04-25T18:58:11.238 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.235+0000 7f4dd464e740 1 Processor -- start 2024-04-25T18:58:11.238 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 -- start start 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a648680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 0x56146a63ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b6af0 con 0x56146a6319b0 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b4330 con 0x56146a63cb90 2024-04-25T18:58:11.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dd464e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b21a0 con 0x56146a6322d0 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 0x56146a63ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.236+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 0x56146a63ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60388/0 (socket says 172.21.15.63:60388) 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dcb400640 1 -- 172.21.15.63:0/285277210 learned_addr learned my addr 172.21.15.63:0/285277210 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47592/0 (socket says 172.21.15.63:47592) 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a648680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 -- 172.21.15.63:0/285277210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 msgr2=0x56146a63ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 0x56146a63ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 -- 172.21.15.63:0/285277210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 msgr2=0x56146a648680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a648680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 -- 172.21.15.63:0/285277210 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56146a629820 con 0x56146a6322d0 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a648680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:11.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dc2000640 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4db40eef90 tx=0x7f4db40f3c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2a197c45361ef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dbb400640 1 -- 172.21.15.63:0/285277210 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db40bdee0 con 0x56146a6322d0 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dbb400640 1 -- 172.21.15.63:0/285277210 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4db4036040 con 0x56146a6322d0 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.237+0000 7f4dbb400640 1 -- 172.21.15.63:0/285277210 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db40bbae0 con 0x56146a6322d0 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 -- 172.21.15.63:0/285277210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 msgr2=0x56146a6343c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4db40eef90 tx=0x7f4db40f3c80 comp rx=0 tx=0).stop 2024-04-25T18:58:11.243 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 -- 172.21.15.63:0/285277210 shutdown_connections 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a63cb90 0x56146a63ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a6322d0 0x56146a6343c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 --2- 172.21.15.63:0/285277210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a648680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.238+0000 7f4dd464e740 1 -- 172.21.15.63:0/285277210 >> 172.21.15.63:0/285277210 conn(0x56146a62dec0 msgr2=0x56146a63b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.239+0000 7f4dd464e740 1 -- 172.21.15.63:0/285277210 shutdown_connections 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.239+0000 7f4dd464e740 1 -- 172.21.15.63:0/285277210 wait complete. 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.239+0000 7f4dd464e740 1 -- start start 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.239+0000 7f4dd464e740 1 Processor -- start 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a6424e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a642f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b27f0 con 0x56146a6319b0 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b3c60 con 0x56146a6322d0 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4dcc1b2ec0 con 0x56146a63cb90 2024-04-25T18:58:11.244 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dd464e740 10 client.0 _ll_register_callbacks cb 0x7f4dbc0044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dcaa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a6424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60402/0 (socket says 172.21.15.63:60402) 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 -- 172.21.15.63:0/760383100 learned_addr learned my addr 172.21.15.63:0/760383100 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dcb400640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a642f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 -- 172.21.15.63:0/760383100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 msgr2=0x56146a642f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a642f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 -- 172.21.15.63:0/760383100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 msgr2=0x56146a6424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a6424e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dc2000640 1 -- 172.21.15.63:0/760383100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56146a6b25b0 con 0x56146a6322d0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.240+0000 7f4dcb400640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a642f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4dc2000640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4db40c5f20 tx=0x7f4db4005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59d6d500e755733 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4db8c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db40c1b90 con 0x56146a6322d0 2024-04-25T18:58:11.245 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7f4db40bbae0 con 0x56146a6322d0 2024-04-25T18:58:11.246 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4db4036040 con 0x56146a6322d0 2024-04-25T18:58:11.246 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f4db40c6450 con 0x56146a6322d0 2024-04-25T18:58:11.246 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4dd464e740 10 client.26555 fetch_fsmap learned FSMap version 45 2024-04-25T18:58:11.246 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56146a6b37c0 con 0x56146a6322d0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.241+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4db4036040 con 0x56146a6322d0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7f4db404d030 con 0x56146a6322d0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 10 client.26555 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 10 client.26555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 20 client.26555 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a6b3eb0 0x56146a6b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 4 client.26555 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56146a5a5320 con 0x56146a6b3eb0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a6b3eb0 0x56146a6b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4dcaa00640 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a6b3eb0 0x56146a6b62d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84fd4bb40ef751fd server_cookie=f33e5fcbbb10e09d in_seq=0 out_seq=0 2024-04-25T18:58:11.247 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.242+0000 7f4db8c00640 10 client.26555 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.287+0000 7f4db8c00640 1 -- 172.21.15.63:0/760383100 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56146a5a5320 con 0x56146a6b3eb0 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.287+0000 7f4db8c00640 10 client.26555 handle_command_reply: tid=0 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.287+0000 7f4dd464e740 1 client.26555 shutdown 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 msgr2=0x56146a642a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4db40c5f20 tx=0x7f4db4005b50 comp rx=0 tx=0).stop 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 shutdown_connections 2024-04-25T18:58:11.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56146a6b3eb0 0x56146a6b62d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56146a6322d0 0x56146a642a20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56146a63cb90 0x56146a642f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4db8c00640 0 client.26555 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 --2- 172.21.15.63:0/760383100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56146a6319b0 0x56146a6424e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 >> 172.21.15.63:0/760383100 conn(0x56146a62dec0 msgr2=0x56146a63bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 shutdown_connections 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.288+0000 7f4dd464e740 1 -- 172.21.15.63:0/760383100 wait complete. 2024-04-25T18:58:11.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.289+0000 7f4dd464e740 20 client.26555 trim_cache size 0 max 16384 2024-04-25T18:58:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:58:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:58:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:58:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:58:11.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.289+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 msgr2=0x7f4db00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.289+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 0x7f4db00a61d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x56146a632f50 tx=0x7f4dbc005f90 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.289+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 msgr2=0x7f4dcc1a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.289+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4db4006010 tx=0x7f4db4030cf0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 shutdown_connections 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f4db00a3d10 0x7f4db00a61d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4dcc112c20 0x7f4dcc19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f4dcc076da0 0x7f4dcc1a4db0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 --2- 172.21.15.63:0/1018041952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4dcc076480 0x7f4dcc1a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 >> 172.21.15.63:0/1018041952 conn(0x7f4dcc0ffa30 msgr2=0x7f4dcc101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 shutdown_connections 2024-04-25T18:58:11.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:11.290+0000 7f4dcbe00640 1 -- 172.21.15.63:0/1018041952 wait complete. 2024-04-25T18:58:11.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 374910 3.41 MB/sec execute 490 sec latency 1169.176 ms 2024-04-25T18:58:12.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 -- 172.21.15.63:0/707480378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec1092d0 msgr2=0x7fb9ec1096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 --2- 172.21.15.63:0/707480378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec1096b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0099c0 tx=0x7fb9dc009cd0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 -- 172.21.15.63:0/707480378 shutdown_connections 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 --2- 172.21.15.63:0/707480378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 --2- 172.21.15.63:0/707480378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec1096b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 --2- 172.21.15.63:0/707480378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec109c80 0x7fb9ec10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 -- 172.21.15.63:0/707480378 >> 172.21.15.63:0/707480378 conn(0x7fb9ec076210 msgr2=0x7fb9ec107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 -- 172.21.15.63:0/707480378 shutdown_connections 2024-04-25T18:58:12.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.505+0000 7fb9f1400700 1 -- 172.21.15.63:0/707480378 wait complete. 2024-04-25T18:58:12.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.506+0000 7fb9f1400700 1 Processor -- start 2024-04-25T18:58:12.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.506+0000 7fb9f1400700 1 -- start start 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec109c80 0x7fb9ec19fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec1a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec119290 con 0x7fb9ec109c80 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec119110 con 0x7fb9ec1092d0 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9f1400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec119410 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60408/0 (socket says 172.21.15.63:60408) 2024-04-25T18:58:12.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9eb200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec1a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9ea800700 1 -- 172.21.15.63:0/673824108 learned_addr learned my addr 172.21.15.63:0/673824108 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:12.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.507+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec109c80 0x7fb9ec19fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 -- 172.21.15.63:0/673824108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 msgr2=0x7fb9ec19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 -- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec109c80 msgr2=0x7fb9ec19fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec109c80 0x7fb9ec19fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 -- 172.21.15.63:0/673824108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9ec1a4670 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9ea800700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9eb200700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec1a3f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400efc0 tx=0x7fb9e400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9e3400700 1 -- 172.21.15.63:0/673824108 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e40092e0 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9e3400700 1 -- 172.21.15.63:0/673824108 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9e4004bb0 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.508+0000 7fb9e3400700 1 -- 172.21.15.63:0/673824108 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4004280 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.509+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9ec1a4900 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.509+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9ec1ac1e0 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376186 3.42 MB/sec execute 491 sec latency 18.244 ms 2024-04-25T18:58:12.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.512+0000 7fb9e3400700 1 -- 172.21.15.63:0/673824108 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb9e401b030 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.514+0000 7fb9fbd01b80 1 Processor -- start 2024-04-25T18:58:12.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.514+0000 7fb9e3400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 0x7fb9d00a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.514+0000 7fb9fbd01b80 1 -- start start 2024-04-25T18:58:12.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.514+0000 7fb9e3400700 1 -- 172.21.15.63:0/673824108 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7fb9e40d0360 con 0x7fb9ec10e3b0 2024-04-25T18:58:12.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 0x7fb9d00a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a34c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a37570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56344597d020 con 0x563445a35180 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634459238c0 con 0x563445a2a2d0 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563445978fe0 con 0x563445a299b0 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47640/0 (socket says 172.21.15.63:47640) 2024-04-25T18:58:12.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9ea800700 1 -- 172.21.15.63:0/4183889930 learned_addr learned my addr 172.21.15.63:0/4183889930 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:12.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.515+0000 7fb9eb200700 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a37570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 -- 172.21.15.63:0/4183889930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 msgr2=0x563445a34c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T18:58:12.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a34c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 -- 172.21.15.63:0/4183889930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 msgr2=0x563445a37570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a37570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 -- 172.21.15.63:0/4183889930 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563445919810 con 0x563445a299b0 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9ea800700 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006010 tx=0x7fb9dc003720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a268f000a528556d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 0x7fb9d00a6310 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec1087c0 tx=0x7fb9d8009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9d7e00700 1 -- 172.21.15.63:0/4183889930 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc027070 con 0x563445a299b0 2024-04-25T18:58:12.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9d7e00700 1 -- 172.21.15.63:0/4183889930 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9dc0037c0 con 0x563445a299b0 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.516+0000 7fb9d7e00700 1 -- 172.21.15.63:0/4183889930 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00baa0 con 0x563445a299b0 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.517+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/4183889930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 msgr2=0x563445a29d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.517+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc006010 tx=0x7fb9dc003720 comp rx=0 tx=0).stop 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.517+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/4183889930 shutdown_connections 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a29d90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a37570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/4183889930 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a34c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/4183889930 >> 172.21.15.63:0/4183889930 conn(0x563445958730 msgr2=0x563445a25e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/4183889930 shutdown_connections 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/4183889930 wait complete. 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.518+0000 7fb9fbd01b80 1 -- start start 2024-04-25T18:58:12.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 Processor -- start 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a31bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a32110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563445a37c40 con 0x563445a299b0 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5634459369f0 con 0x563445a35180 2024-04-25T18:58:12.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563445a37dc0 con 0x563445a2a2d0 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.519+0000 7fb9fbd01b80 10 client.0 _ll_register_callbacks cb 0x5634459158b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a31bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47094/0 (socket says 172.21.15.63:47094) 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9ea800700 1 -- 172.21.15.63:0/1114417641 learned_addr learned my addr 172.21.15.63:0/1114417641 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9eb200700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a32110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 msgr2=0x563445a32110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a32110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1114417641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 msgr2=0x563445a348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1114417641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634458caa60 con 0x563445a2a2d0 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.520+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a31bd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec06c260 tx=0x7fb9d80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a5cb96c91b18226 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8004230 con 0x563445a2a2d0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9dc0037c0 con 0x563445a2a2d0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7fb9d80043d0 con 0x563445a2a2d0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb9d801c830 con 0x563445a2a2d0 2024-04-25T18:58:12.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9fbd01b80 10 client.26584 fetch_fsmap learned FSMap version 45 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563445913c80 con 0x563445a2a2d0 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.521+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9dc0037c0 con 0x563445a2a2d0 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fb9d8028430 con 0x563445a2a2d0 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 10 client.26584 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 10 client.26584 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 20 client.26584 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:12.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445a449e0 0x563445a46e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 4 client.26584 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56344598a810 con 0x563445a449e0 2024-04-25T18:58:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.522+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445a449e0 0x563445a46e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.523+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445a449e0 0x563445a46e80 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79d5eaa22ee8bfb1 server_cookie=dece4e1279997de1 in_seq=0 out_seq=0 2024-04-25T18:58:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.523+0000 7fb9d4c00700 10 client.26584 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1114417641 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56344598a810 con 0x563445a449e0 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9d4c00700 10 client.26584 handle_command_reply: tid=0 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 client.26584 shutdown 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 msgr2=0x563445a31bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a31bd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec06c260 tx=0x7fb9d80047c0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 shutdown_connections 2024-04-25T18:58:12.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445a449e0 0x563445a46e80 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a32110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9d4c00700 0 client.26584 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1114417641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a31bd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 >> 172.21.15.63:0/1114417641 conn(0x563445958730 msgr2=0x563445a25a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 shutdown_connections 2024-04-25T18:58:12.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1114417641 wait complete. 2024-04-25T18:58:12.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.524+0000 7fb9fbd01b80 20 client.26584 trim_cache size 0 max 16384 2024-04-25T18:58:12.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 Processor -- start 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 -- start start 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a3d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b6270 con 0x563445a35180 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b3a70 con 0x563445a2a2d0 2024-04-25T18:58:12.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.548+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b18e0 con 0x563445a299b0 2024-04-25T18:58:12.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47652/0 (socket says 172.21.15.63:47652) 2024-04-25T18:58:12.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 -- 172.21.15.63:0/2430050949 learned_addr learned my addr 172.21.15.63:0/2430050949 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:12.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9eb200700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a3d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 -- 172.21.15.63:0/2430050949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 msgr2=0x563445a40b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a40b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 -- 172.21.15.63:0/2430050949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 msgr2=0x563445a3d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a3d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 -- 172.21.15.63:0/2430050949 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563445a30180 con 0x563445a299b0 2024-04-25T18:58:12.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9eb200700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.549+0000 7fb9ea800700 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc003e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f7b24805d47130f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.550+0000 7fb9d7400700 1 -- 172.21.15.63:0/2430050949 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc027040 con 0x563445a299b0 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.550+0000 7fb9d7400700 1 -- 172.21.15.63:0/2430050949 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9dc036070 con 0x563445a299b0 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.550+0000 7fb9d7400700 1 -- 172.21.15.63:0/2430050949 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc031100 con 0x563445a299b0 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/2430050949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 msgr2=0x563445a25c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc003e20 comp rx=0 tx=0).stop 2024-04-25T18:58:12.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/2430050949 shutdown_connections 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a299b0 0x563445a25c00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a35180 0x563445a3d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/2430050949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a2a2d0 0x563445a40b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/2430050949 >> 172.21.15.63:0/2430050949 conn(0x563445958730 msgr2=0x563445a33400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/2430050949 shutdown_connections 2024-04-25T18:58:12.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/2430050949 wait complete. 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.551+0000 7fb9fbd01b80 1 -- start start 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 Processor -- start 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a3bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b1f30 con 0x563445a299b0 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b33a0 con 0x563445a35180 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb9ec1b2600 con 0x563445a2a2d0 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.552+0000 7fb9fbd01b80 10 client.0 _ll_register_callbacks cb 0x5634459158b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47654/0 (socket says 172.21.15.63:47654) 2024-04-25T18:58:12.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1163673608 learned_addr learned my addr 172.21.15.63:0/1163673608 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:12.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9eb200700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a3bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 msgr2=0x563445a3bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a3bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1163673608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 msgr2=0x563445a3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 -- 172.21.15.63:0/1163673608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563445a3c150 con 0x563445a2a2d0 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9e9e00700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800efd0 tx=0x7fb9d8028dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adce283cbc315675 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.553+0000 7fb9d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d8034040 con 0x563445a2a2d0 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 8631+0+0 (secure 0 0 0) 0x7fb9d801c9d0 con 0x563445a2a2d0 2024-04-25T18:58:12.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9dc036070 con 0x563445a2a2d0 2024-04-25T18:58:12.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb9d8045050 con 0x563445a2a2d0 2024-04-25T18:58:12.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9fbd01b80 10 client.26590 fetch_fsmap learned FSMap version 45 2024-04-25T18:58:12.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x563445aabe80 con 0x563445a2a2d0 2024-04-25T18:58:12.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.554+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9dc036070 con 0x563445a2a2d0 2024-04-25T18:58:12.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 45) ==== 8358+0+0 (secure 0 0 0) 0x7fb9d8040030 con 0x563445a2a2d0 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 10 client.26590 fetch_fsmap finished waiting for FSMap version 45 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 10 client.26590 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 20 client.26590 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445aac5c0 0x563445aae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 4 client.26590 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.5, "await_for": 42.0}] 2024-04-25T18:58:12.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.5, "await_for": 42.0}) -- 0x5634459848b0 con 0x563445aac5c0 2024-04-25T18:58:12.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445aac5c0 0x563445aae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:12.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9ea800700 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445aac5c0 0x563445aae9e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6472e6eb5562b035 server_cookie=847eee4cc0a45b10 in_seq=0 out_seq=0 2024-04-25T18:58:12.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.555+0000 7fb9d4c00700 10 client.26590 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.637+0000 7fb9d4c00700 1 -- 172.21.15.63:0/1163673608 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5634459848b0 con 0x563445aac5c0 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9d4c00700 10 client.26590 handle_command_reply: tid=0 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 client.26590 shutdown 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 msgr2=0x563445a3b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800efd0 tx=0x7fb9d8028dc0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 shutdown_connections 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563445aac5c0 0x563445aae9e0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563445a35180 0x563445a3bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563445a299b0 0x563445a3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 --2- 172.21.15.63:0/1163673608 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563445a2a2d0 0x563445a3b5d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9d4c00700 0 client.26590 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 >> 172.21.15.63:0/1163673608 conn(0x563445958730 msgr2=0x563445a33c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 shutdown_connections 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.638+0000 7fb9fbd01b80 1 -- 172.21.15.63:0/1163673608 wait complete. 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.639+0000 7fb9fbd01b80 20 client.26590 trim_cache size 0 max 16384 2024-04-25T18:58:12.642 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 45, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 34, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "eef41991": { 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 34, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 81.5, 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:58:12.643 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:12.644 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 msgr2=0x7fb9d00a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 0x7fb9d00a6310 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec1087c0 tx=0x7fb9d8009260 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 msgr2=0x7fb9ec1a3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec1a3f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400efc0 tx=0x7fb9e400c3a0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 shutdown_connections 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb9d00a3e50 0x7fb9d00a6310 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb9ec10e3b0 0x7fb9ec1a3f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb9ec1092d0 0x7fb9ec19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 --2- 172.21.15.63:0/673824108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb9ec109c80 0x7fb9ec19fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 >> 172.21.15.63:0/673824108 conn(0x7fb9ec076210 msgr2=0x7fb9ec076790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 shutdown_connections 2024-04-25T18:58:12.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:12.640+0000 7fb9f1400700 1 -- 172.21.15.63:0/673824108 wait complete. 2024-04-25T18:58:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:12 smithi070 ceph-mon[32820]: pgmap v365: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:12.714 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:12 smithi063 ceph-mon[28006]: pgmap v365: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:12 smithi114 ceph-mon[32988]: pgmap v365: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:13.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.42 MB/sec execute 492 sec latency 934.405 ms 2024-04-25T18:58:13.782 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'eef41991', quiesce took 8.8 seconds. Will release after: 21.5 2024-04-25T18:58:14.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.41 MB/sec execute 493 sec latency 1934.542 ms 2024-04-25T18:58:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:14 smithi070 ceph-mon[32820]: pgmap v366: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:14 smithi070 ceph-mon[32820]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T18:58:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:14 smithi114 ceph-mon[32988]: pgmap v366: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:14 smithi114 ceph-mon[32988]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T18:58:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:14 smithi063 ceph-mon[28006]: pgmap v366: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 177 op/s 2024-04-25T18:58:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:14 smithi063 ceph-mon[28006]: osdmap e111: 12 total, 12 up, 12 in 2024-04-25T18:58:15.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.40 MB/sec execute 494 sec latency 2934.701 ms 2024-04-25T18:58:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:16 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:16.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:16 smithi114 ceph-mon[32988]: pgmap v368: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:16 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:16 smithi063 ceph-mon[28006]: pgmap v368: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:16 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:16 smithi070 ceph-mon[32820]: pgmap v368: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:16.513 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.39 MB/sec execute 495 sec latency 3934.836 ms 2024-04-25T18:58:17.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.39 MB/sec execute 496 sec latency 4934.987 ms 2024-04-25T18:58:18.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.38 MB/sec execute 497 sec latency 5935.137 ms 2024-04-25T18:58:18.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:18 smithi114 ceph-mon[32988]: pgmap v369: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:18.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:18 smithi063 ceph-mon[28006]: pgmap v369: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:18 smithi070 ceph-mon[32820]: pgmap v369: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 893 KiB/s wr, 133 op/s 2024-04-25T18:58:19.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.37 MB/sec execute 498 sec latency 6935.261 ms 2024-04-25T18:58:20.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.37 MB/sec execute 499 sec latency 7935.341 ms 2024-04-25T18:58:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:20 smithi114 ceph-mon[32988]: pgmap v370: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:20 smithi063 ceph-mon[28006]: pgmap v370: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:20.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:20 smithi070 ceph-mon[32820]: pgmap v370: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:21.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.36 MB/sec execute 500 sec latency 8935.492 ms 2024-04-25T18:58:22.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.35 MB/sec execute 501 sec latency 9935.659 ms 2024-04-25T18:58:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:22 smithi114 ceph-mon[32988]: pgmap v371: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:22 smithi063 ceph-mon[28006]: pgmap v371: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:22 smithi070 ceph-mon[32820]: pgmap v371: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:23.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.35 MB/sec execute 502 sec latency 10935.751 ms 2024-04-25T18:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.438+0000 7fcc5e200640 1 -- 172.21.15.63:0/290204100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc58102990 msgr2=0x7fcc581069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.438+0000 7fcc5e200640 1 --2- 172.21.15.63:0/290204100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc58102990 0x7fcc581069d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009900 tx=0x7fcc4002a3f0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 -- 172.21.15.63:0/290204100 shutdown_connections 2024-04-25T18:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 --2- 172.21.15.63:0/290204100 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc581070b0 0x7fcc5810ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 --2- 172.21.15.63:0/290204100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc58102990 0x7fcc581069d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 --2- 172.21.15.63:0/290204100 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc58101fe0 0x7fcc581023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 -- 172.21.15.63:0/290204100 >> 172.21.15.63:0/290204100 conn(0x7fcc580fd8a0 msgr2=0x7fcc580ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 -- 172.21.15.63:0/290204100 shutdown_connections 2024-04-25T18:58:24.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.439+0000 7fcc5e200640 1 -- 172.21.15.63:0/290204100 wait complete. 2024-04-25T18:58:24.442 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.440+0000 7fcc5e200640 1 Processor -- start 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.440+0000 7fcc5e200640 1 -- start start 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc58101fe0 0x7fcc581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 0x7fcc58198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 0x7fcc5819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc58111ff0 con 0x7fcc581070b0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc58111e70 con 0x7fcc5819cea0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5e200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc58112170 con 0x7fcc58101fe0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 0x7fcc58198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc58101fe0 0x7fcc581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 0x7fcc58198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50904/0 (socket says 172.21.15.63:50904) 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc4fe00640 1 -- 172.21.15.63:0/2094615030 learned_addr learned my addr 172.21.15.63:0/2094615030 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 0x7fcc5819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 -- 172.21.15.63:0/2094615030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc58101fe0 msgr2=0x7fcc581985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc58101fe0 0x7fcc581985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 -- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 msgr2=0x7fcc58198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 0x7fcc58198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.441+0000 7fcc5d800640 1 -- 172.21.15.63:0/2094615030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5819d990 con 0x7fcc5819cea0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 0x7fcc5819d350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009fd0 tx=0x7fcc5000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc56000640 1 -- 172.21.15.63:0/2094615030 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000ce20 con 0x7fcc5819cea0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5819dc80 con 0x7fcc5819cea0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc56000640 1 -- 172.21.15.63:0/2094615030 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc50005020 con 0x7fcc5819cea0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc56000640 1 -- 172.21.15.63:0/2094615030 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50010690 con 0x7fcc5819cea0 2024-04-25T18:58:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.442+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc581a5470 con 0x7fcc5819cea0 2024-04-25T18:58:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.445+0000 7fcc625f3740 1 Processor -- start 2024-04-25T18:58:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc56000640 1 -- 172.21.15.63:0/2094615030 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcc500054d0 con 0x7fcc5819cea0 2024-04-25T18:58:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 -- start start 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 0x55728c487a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c48a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55728c3de900 con 0x55728c478870 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55728c384ef0 con 0x55728c487fa0 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.446+0000 7fcc625f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55728c3da8c0 con 0x55728c479190 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50914/0 (socket says 172.21.15.63:50914) 2024-04-25T18:58:24.449 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc4fe00640 1 -- 172.21.15.63:0/1281297281 learned_addr learned my addr 172.21.15.63:0/1281297281 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 0x55728c487a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c48a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 -- 172.21.15.63:0/1281297281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 msgr2=0x55728c48a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c48a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 -- 172.21.15.63:0/1281297281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 msgr2=0x55728c478c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 -- 172.21.15.63:0/1281297281 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c0f1210 con 0x55728c479190 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.447+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 0x55728c487a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c20 tx=0x7fcc48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6689a7039442c2ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc56000640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 0x7fcc380a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc4e000640 1 -- 172.21.15.63:0/1281297281 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48009dd0 con 0x55728c479190 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc4e000640 1 -- 172.21.15.63:0/1281297281 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc48005020 con 0x55728c479190 2024-04-25T18:58:24.450 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc56000640 1 -- 172.21.15.63:0/2094615030 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcc500d0f00 con 0x7fcc5819cea0 2024-04-25T18:58:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc4e000640 1 -- 172.21.15.63:0/1281297281 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48004b20 con 0x55728c479190 2024-04-25T18:58:24.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 0x7fcc380a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc625f3740 1 -- 172.21.15.63:0/1281297281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 msgr2=0x55728c487a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.448+0000 7fcc625f3740 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 0x55728c487a60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c20 tx=0x7fcc48011810 comp rx=0 tx=0).stop 2024-04-25T18:58:24.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 -- 172.21.15.63:0/1281297281 shutdown_connections 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c48a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 0x7fcc380a61d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005c40 tx=0x7fcc40005b50 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c479190 0x55728c487a60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 --2- 172.21.15.63:0/1281297281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c478c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 -- 172.21.15.63:0/1281297281 >> 172.21.15.63:0/1281297281 conn(0x55728c474ec0 msgr2=0x55728c4752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 -- 172.21.15.63:0/1281297281 shutdown_connections 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 -- 172.21.15.63:0/1281297281 wait complete. 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 -- start start 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.449+0000 7fcc625f3740 1 Processor -- start 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c484f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c485460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 0x55728c4859a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.455 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55728c48ab70 con 0x55728c487fa0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55728c398310 con 0x55728c479190 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55728c48acf0 con 0x55728c478870 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc625f3740 10 client.0 _ll_register_callbacks cb 0x55728c366cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c484f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c485460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc5d800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 0x55728c4859a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c484f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46368/0 (socket says 172.21.15.63:46368) 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.450+0000 7fcc5ce00640 1 -- 172.21.15.63:0/2704005146 learned_addr learned my addr 172.21.15.63:0/2704005146 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 -- 172.21.15.63:0/2704005146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 msgr2=0x55728c484f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c484f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 -- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 msgr2=0x55728c485460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c485460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 -- 172.21.15.63:0/2704005146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c1bd620 con 0x55728c487fa0 2024-04-25T18:58:24.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c485460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 0x55728c4859a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc500ce5b0 tx=0x7fcc5009ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5d5fc35900216e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc46a00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50099470 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcc500a55e0 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.451+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc48005020 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fcc50099610 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 10 client.16395 fetch_fsmap learned FSMap version 46 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55728c496e90 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc48005020 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7fcc50098c70 con 0x55728c487fa0 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 10 client.16395 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 20 client.16395 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c498350 0x55728c49a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 4 client.16395 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.452+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55728c3f28b0 con 0x55728c498350 2024-04-25T18:58:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.453+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c498350 0x55728c49a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.453+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c498350 0x55728c49a740 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ce4724a60a62b server_cookie=170314efba9b36a2 in_seq=0 out_seq=0 2024-04-25T18:58:24.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.453+0000 7fcc46a00640 10 client.16395 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.454+0000 7fcc46a00640 1 -- 172.21.15.63:0/2704005146 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55728c3f28b0 con 0x55728c498350 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.454+0000 7fcc46a00640 10 client.16395 handle_command_reply: tid=0 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.454+0000 7fcc625f3740 1 client.16395 shutdown 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.454+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 msgr2=0x55728c4859a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.454+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 0x55728c4859a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcc500ce5b0 tx=0x7fcc5009ce00 comp rx=0 tx=0).stop 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 shutdown_connections 2024-04-25T18:58:24.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c498350 0x55728c49a740 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c485460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c487fa0 0x55728c4859a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2704005146 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c484f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc46a00640 0 client.16395 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 >> 172.21.15.63:0/2704005146 conn(0x55728c474ec0 msgr2=0x55728c47ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 shutdown_connections 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 1 -- 172.21.15.63:0/2704005146 wait complete. 2024-04-25T18:58:24.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.455+0000 7fcc625f3740 20 client.16395 trim_cache size 0 max 16384 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.471+0000 7fcc625f3740 1 Processor -- start 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.471+0000 7fcc625f3740 1 -- start start 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c4a0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c487fa0 0x55728c4a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581ae2b0 con 0x55728c478870 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581abb20 con 0x55728c479190 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc625f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581a9990 con 0x55728c487fa0 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c4a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50934/0 (socket says 172.21.15.63:50934) 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc5ce00640 1 -- 172.21.15.63:0/2204877156 learned_addr learned my addr 172.21.15.63:0/2204877156 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.472+0000 7fcc5d800640 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c487fa0 0x55728c4a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 -- 172.21.15.63:0/2204877156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c487fa0 msgr2=0x55728c4a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c487fa0 0x55728c4a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 -- 172.21.15.63:0/2204877156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 msgr2=0x55728c4a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c4a0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 -- 172.21.15.63:0/2204877156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55728c494f40 con 0x55728c478870 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c4a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c20 tx=0x7fcc48009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2e0ed0cce75413 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc4d600640 1 -- 172.21.15.63:0/2204877156 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48031070 con 0x55728c478870 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc4d600640 1 -- 172.21.15.63:0/2204877156 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4802b2d0 con 0x55728c478870 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.473+0000 7fcc4d600640 1 -- 172.21.15.63:0/2204877156 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc48024410 con 0x55728c478870 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- 172.21.15.63:0/2204877156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 msgr2=0x55728c493a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c20 tx=0x7fcc48009820 comp rx=0 tx=0).stop 2024-04-25T18:58:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- 172.21.15.63:0/2204877156 shutdown_connections 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c479190 0x55728c4a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c487fa0 0x55728c4a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 --2- 172.21.15.63:0/2204877156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c478870 0x55728c493a70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- 172.21.15.63:0/2204877156 >> 172.21.15.63:0/2204877156 conn(0x55728c474ec0 msgr2=0x55728c480280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- 172.21.15.63:0/2204877156 shutdown_connections 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- 172.21.15.63:0/2204877156 wait complete. 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.474+0000 7fcc625f3740 1 -- start start 2024-04-25T18:58:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 Processor -- start 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c48d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c4fc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581a9fe0 con 0x55728c479190 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581ab450 con 0x55728c487fa0 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcc581aa6b0 con 0x55728c478870 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.475+0000 7fcc625f3740 10 client.0 _ll_register_callbacks cb 0x7fcc5009c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c4fc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc4fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50944/0 (socket says 172.21.15.63:50944) 2024-04-25T18:58:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc4fe00640 1 -- 172.21.15.63:0/604246127 learned_addr learned my addr 172.21.15.63:0/604246127 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c48d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 -- 172.21.15.63:0/604246127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 msgr2=0x55728c48d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c48d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 -- 172.21.15.63:0/604246127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 msgr2=0x55728c48d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 -- 172.21.15.63:0/604246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55728c4fce00 con 0x55728c487fa0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc4fe00640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc5d800640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c4fc8c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcc500a5540 tx=0x7fcc50005870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc45bdfcdc64d491 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc46a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.476+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc50099470 con 0x55728c487fa0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc4802b2d0 con 0x55728c487fa0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcc500a57f0 con 0x55728c487fa0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fcc5009cbd0 con 0x55728c487fa0 2024-04-25T18:58:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 10 client.26588 fetch_fsmap learned FSMap version 46 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55728c4fe040 con 0x55728c487fa0 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc4802b2d0 con 0x55728c487fa0 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 46) ==== 8354+0+0 (secure 0 0 0) 0x7fcc500a2390 con 0x55728c487fa0 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 10 client.26588 fetch_fsmap finished waiting for FSMap version 46 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 10 client.26588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 20 client.26588 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c4fe810 0x55728c500c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 4 client.26588 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55728c3ec320 con 0x55728c4fe810 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.477+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c4fe810 0x55728c500c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.478+0000 7fcc5ce00640 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c4fe810 0x55728c500c30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0da4ea175609b7 server_cookie=ccfd2b054e35a2d9 in_seq=0 out_seq=0 2024-04-25T18:58:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.478+0000 7fcc46a00640 10 client.26588 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc46a00640 1 -- 172.21.15.63:0/604246127 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55728c3ec320 con 0x55728c4fe810 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc46a00640 10 client.26588 handle_command_reply: tid=0 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 client.26588 shutdown 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 msgr2=0x55728c4fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c4fc8c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcc500a5540 tx=0x7fcc50005870 comp rx=0 tx=0).stop 2024-04-25T18:58:24.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 shutdown_connections 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55728c4fe810 0x55728c500c30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55728c487fa0 0x55728c4fc8c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55728c479190 0x55728c48d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 --2- 172.21.15.63:0/604246127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55728c478870 0x55728c48d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.511+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 >> 172.21.15.63:0/604246127 conn(0x55728c474ec0 msgr2=0x55728c473790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.512+0000 7fcc46a00640 0 client.26588 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.512+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 shutdown_connections 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.512+0000 7fcc625f3740 1 -- 172.21.15.63:0/604246127 wait complete. 2024-04-25T18:58:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.512+0000 7fcc625f3740 20 client.26588 trim_cache size 0 max 16384 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 msgr2=0x7fcc380a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 0x7fcc380a61d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005c40 tx=0x7fcc40005b50 comp rx=0 tx=0).stop 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 msgr2=0x7fcc5819d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 0x7fcc5819d350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcc50009fd0 tx=0x7fcc5000ef10 comp rx=0 tx=0).stop 2024-04-25T18:58:24.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.34 MB/sec execute 503 sec latency 11935.841 ms 2024-04-25T18:58:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 shutdown_connections 2024-04-25T18:58:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcc380a3d10 0x7fcc380a61d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc5819cea0 0x7fcc5819d350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc581070b0 0x7fcc58198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 --2- 172.21.15.63:0/2094615030 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcc58101fe0 0x7fcc581985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 >> 172.21.15.63:0/2094615030 conn(0x7fcc580fd8a0 msgr2=0x7fcc580fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.513+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 shutdown_connections 2024-04-25T18:58:24.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:24.514+0000 7fcc5e200640 1 -- 172.21.15.63:0/2094615030 wait complete. 2024-04-25T18:58:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:24 smithi114 ceph-mon[32988]: pgmap v372: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:24 smithi063 ceph-mon[28006]: pgmap v372: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:24 smithi070 ceph-mon[32820]: pgmap v372: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 623 KiB/s wr, 85 op/s 2024-04-25T18:58:25.514 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.33 MB/sec execute 504 sec latency 12935.924 ms 2024-04-25T18:58:26.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.33 MB/sec execute 505 sec latency 13936.080 ms 2024-04-25T18:58:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:26 smithi114 ceph-mon[32988]: pgmap v373: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 786 KiB/s wr, 65 op/s 2024-04-25T18:58:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:26 smithi063 ceph-mon[28006]: pgmap v373: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 786 KiB/s wr, 65 op/s 2024-04-25T18:58:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:26 smithi070 ceph-mon[32820]: pgmap v373: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 786 KiB/s wr, 65 op/s 2024-04-25T18:58:27.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.32 MB/sec execute 506 sec latency 14936.219 ms 2024-04-25T18:58:28.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.31 MB/sec execute 507 sec latency 15936.362 ms 2024-04-25T18:58:28.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:28 smithi114 ceph-mon[32988]: pgmap v374: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 580 KiB/s wr, 35 op/s 2024-04-25T18:58:28.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:28 smithi063 ceph-mon[28006]: pgmap v374: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 580 KiB/s wr, 35 op/s 2024-04-25T18:58:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:28 smithi070 ceph-mon[32820]: pgmap v374: 97 pgs: 97 active+clean; 114 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 580 KiB/s wr, 35 op/s 2024-04-25T18:58:29.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.31 MB/sec execute 508 sec latency 16936.523 ms 2024-04-25T18:58:30.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.30 MB/sec execute 509 sec latency 17936.682 ms 2024-04-25T18:58:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:30 smithi114 ceph-mon[32988]: pgmap v375: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 919 KiB/s wr, 57 op/s 2024-04-25T18:58:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:30 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:30.591 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:30 smithi063 ceph-mon[28006]: pgmap v375: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 919 KiB/s wr, 57 op/s 2024-04-25T18:58:30.591 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:30 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:30 smithi070 ceph-mon[32820]: pgmap v375: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 919 KiB/s wr, 57 op/s 2024-04-25T18:58:30.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:30 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:31.515 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.29 MB/sec execute 510 sec latency 18936.825 ms 2024-04-25T18:58:32.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.29 MB/sec execute 511 sec latency 19936.986 ms 2024-04-25T18:58:32.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:32 smithi063 ceph-mon[28006]: pgmap v376: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:32.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:32 smithi070 ceph-mon[32820]: pgmap v376: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:32 smithi114 ceph-mon[32988]: pgmap v376: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:33.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.28 MB/sec execute 512 sec latency 20937.132 ms 2024-04-25T18:58:34.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.28 MB/sec execute 513 sec latency 21937.336 ms 2024-04-25T18:58:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:34 smithi063 ceph-mon[28006]: pgmap v377: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:34.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:34 smithi070 ceph-mon[32820]: pgmap v377: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:34 smithi114 ceph-mon[32988]: pgmap v377: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 643 KiB/s wr, 41 op/s 2024-04-25T18:58:35.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:58:35.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.27 MB/sec execute 514 sec latency 22937.434 ms 2024-04-25T18:58:35.764 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:36.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.26 MB/sec execute 515 sec latency 23937.530 ms 2024-04-25T18:58:36.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:36 smithi063 ceph-mon[28006]: pgmap v378: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 901 KiB/s wr, 56 op/s 2024-04-25T18:58:36.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:36 smithi070 ceph-mon[32820]: pgmap v378: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 901 KiB/s wr, 56 op/s 2024-04-25T18:58:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:36 smithi114 ceph-mon[32988]: pgmap v378: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 901 KiB/s wr, 56 op/s 2024-04-25T18:58:37.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.26 MB/sec execute 516 sec latency 24937.611 ms 2024-04-25T18:58:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3367165650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108490 msgr2=0x7fcfe81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3367165650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108490 0x7fcfe81088d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009cd0 tx=0x7fcfd002a8e0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3367165650 shutdown_connections 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3367165650 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108e10 0x7fcfe8115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3367165650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108490 0x7fcfe81088d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3367165650 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 0x7fcfe810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3367165650 >> 172.21.15.63:0/3367165650 conn(0x7fcfe8077d30 msgr2=0x7fcfe8106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3367165650 shutdown_connections 2024-04-25T18:58:37.658 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.656+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3367165650 wait complete. 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.657+0000 7fcfe7e00640 1 Processor -- start 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.657+0000 7fcfe7e00640 1 -- start start 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108e10 0x7fcfe81a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 0x7fcfe819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8118640 con 0x7fcfe8108e10 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81184c0 con 0x7fcfe8108490 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81187c0 con 0x7fcfe810e420 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56386/0 (socket says 172.21.15.63:56386) 2024-04-25T18:58:37.660 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfde000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108e10 0x7fcfe81a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 -- 172.21.15.63:0/3069268600 learned_addr learned my addr 172.21.15.63:0/3069268600 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 0x7fcfe819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 -- 172.21.15.63:0/3069268600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 msgr2=0x7fcfe819e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 0x7fcfe819e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 -- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108e10 msgr2=0x7fcfe81a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108e10 0x7fcfe81a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.658+0000 7fcfe6a00640 1 -- 172.21.15.63:0/3069268600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe819f140 con 0x7fcfe8108490 2024-04-25T18:58:37.661 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e9f0 tx=0x7fcfd800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfdfe00640 1 -- 172.21.15.63:0/3069268600 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800cd80 con 0x7fcfe8108490 2024-04-25T18:58:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfdfe00640 1 -- 172.21.15.63:0/3069268600 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd80054d0 con 0x7fcfe8108490 2024-04-25T18:58:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe819f430 con 0x7fcfe8108490 2024-04-25T18:58:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfdfe00640 1 -- 172.21.15.63:0/3069268600 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8010690 con 0x7fcfe8108490 2024-04-25T18:58:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.659+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe81ab6d0 con 0x7fcfe8108490 2024-04-25T18:58:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcfdfe00640 1 -- 172.21.15.63:0/3069268600 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fcfd8004e90 con 0x7fcfe8108490 2024-04-25T18:58:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 Processor -- start 2024-04-25T18:58:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 -- start start 2024-04-25T18:58:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a03684a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a03684cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a0367a5900 con 0x55a03684ab90 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a03674bef0 con 0x55a0368402d0 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.663+0000 7fcff087c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a0367a18c0 con 0x55a03683f9b0 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33058/0 (socket says 172.21.15.63:33058) 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 -- 172.21.15.63:0/2947459569 learned_addr learned my addr 172.21.15.63:0/2947459569 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a03684a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfde000640 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a03684cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 -- 172.21.15.63:0/2947459569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 msgr2=0x55a03684a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a03684a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 -- 172.21.15.63:0/2947459569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 msgr2=0x55a03684cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a03684cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfe7400640 1 -- 172.21.15.63:0/2947459569 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0364b8210 con 0x55a03683f9b0 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.664+0000 7fcfdfe00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 0x7fcfcc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfe7400640 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009c00 tx=0x7fcfe00117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb8841cf6f25fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfdfe00640 1 -- 172.21.15.63:0/3069268600 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcfd80d0cf0 con 0x7fcfe8108490 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 0x7fcfcc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfd7e00640 1 -- 172.21.15.63:0/2947459569 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0012450 con 0x55a03683f9b0 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfd7e00640 1 -- 172.21.15.63:0/2947459569 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfe0012aa0 con 0x55a03683f9b0 2024-04-25T18:58:37.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.665+0000 7fcfd7e00640 1 -- 172.21.15.63:0/2947459569 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe000fae0 con 0x55a03683f9b0 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- 172.21.15.63:0/2947459569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 msgr2=0x55a03683fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009c00 tx=0x7fcfe00117b0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcfe6a00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 0x7fcfcc0a61d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a036840f50 tx=0x7fcfd80d47a0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- 172.21.15.63:0/2947459569 shutdown_connections 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a03684a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a03684cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 --2- 172.21.15.63:0/2947459569 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03683f9b0 0x55a03683fd90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- 172.21.15.63:0/2947459569 >> 172.21.15.63:0/2947459569 conn(0x55a03683bec0 msgr2=0x55a03683c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- 172.21.15.63:0/2947459569 shutdown_connections 2024-04-25T18:58:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- 172.21.15.63:0/2947459569 wait complete. 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.666+0000 7fcff087c740 1 -- start start 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 Processor -- start 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a036848010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03684ab90 0x55a036848550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a03684d7c0 con 0x55a03683f9b0 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a03675f310 con 0x55a03684ab90 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a03684d940 con 0x55a0368402d0 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcff087c740 10 client.0 _ll_register_callbacks cb 0x55a03672dcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcfe7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcfe7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42944/0 (socket says 172.21.15.63:42944) 2024-04-25T18:58:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcfe7400640 1 -- 172.21.15.63:0/3224342851 learned_addr learned my addr 172.21.15.63:0/3224342851 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.667+0000 7fcfde000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a036848010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03684ab90 0x55a036848550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 -- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03684ab90 msgr2=0x55a036848550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03684ab90 0x55a036848550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 -- 172.21.15.63:0/3224342851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 msgr2=0x55a036847ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 -- 172.21.15.63:0/3224342851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a036584620 con 0x55a0368402d0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfde000640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a036848010 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0000f80 tx=0x7fcfd0031a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56c21f54ea26ed29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.668+0000 7fcfd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0048020 con 0x55a0368402d0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe0012aa0 con 0x55a0368402d0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcfd0003970 con 0x55a0368402d0 2024-04-25T18:58:37.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcfd0003b70 con 0x55a0368402d0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcff087c740 10 client.26620 fetch_fsmap learned FSMap version 47 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a036859a20 con 0x55a0368402d0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfe0012aa0 con 0x55a0368402d0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.669+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7fcfd0043070 con 0x55a0368402d0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 10 client.26620 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 10 client.26620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 20 client.26620 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a03685ae90 0x55a03685d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 4 client.26620 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0367b98b0 con 0x55a03685ae90 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.670+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a03685ae90 0x55a03685d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.671+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a03685ae90 0x55a03685d2c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d84004834690f3 server_cookie=6aea10efae5529a4 in_seq=0 out_seq=0 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.671+0000 7fcfd4c00640 10 client.26620 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:37.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.671+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3224342851 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0367b98b0 con 0x55a03685ae90 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.671+0000 7fcfd4c00640 10 client.26620 handle_command_reply: tid=0 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.671+0000 7fcff087c740 1 client.26620 shutdown 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 msgr2=0x55a036848010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a036848010 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0000f80 tx=0x7fcfd0031a80 comp rx=0 tx=0).stop 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 shutdown_connections 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a03685ae90 0x55a03685d2c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03684ab90 0x55a036848550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a036848010 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcfd4c00640 0 client.26620 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 --2- 172.21.15.63:0/3224342851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036847ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 >> 172.21.15.63:0/3224342851 conn(0x55a03683bec0 msgr2=0x55a036841bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 shutdown_connections 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 1 -- 172.21.15.63:0/3224342851 wait complete. 2024-04-25T18:58:37.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.672+0000 7fcff087c740 20 client.26620 trim_cache size 0 max 16384 2024-04-25T18:58:37.690 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.688+0000 7fcff087c740 1 Processor -- start 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 -- start start 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036856680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03684ab90 0x55a036863110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b6a30 con 0x55a03683f9b0 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b4270 con 0x55a0368402d0 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcff087c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b20e0 con 0x55a03684ab90 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcfe7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036856680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcfde000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcfde000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56404/0 (socket says 172.21.15.63:56404) 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcfde000640 1 -- 172.21.15.63:0/2761441431 learned_addr learned my addr 172.21.15.63:0/2761441431 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.689+0000 7fcfe7400640 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03684ab90 0x55a036863110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.691 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 -- 172.21.15.63:0/2761441431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03684ab90 msgr2=0x55a036863110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03684ab90 0x55a036863110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 -- 172.21.15.63:0/2761441431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 msgr2=0x55a036856680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036856680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.692 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 -- 172.21.15.63:0/2761441431 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a036840100 con 0x55a0368402d0 2024-04-25T18:58:37.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfde000640 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0030790 tx=0x7fcfd0009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a961ae31726269be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfd7400640 1 -- 172.21.15.63:0/2761441431 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0048070 con 0x55a0368402d0 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfd7400640 1 -- 172.21.15.63:0/2761441431 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd0036040 con 0x55a0368402d0 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.690+0000 7fcfd7400640 1 -- 172.21.15.63:0/2761441431 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00309f0 con 0x55a0368402d0 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- 172.21.15.63:0/2761441431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 msgr2=0x55a036862bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0030790 tx=0x7fcfd0009540 comp rx=0 tx=0).stop 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- 172.21.15.63:0/2761441431 shutdown_connections 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a0368402d0 0x55a036862bd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a03684ab90 0x55a036863110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 --2- 172.21.15.63:0/2761441431 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03683f9b0 0x55a036856680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- 172.21.15.63:0/2761441431 >> 172.21.15.63:0/2761441431 conn(0x55a03683bec0 msgr2=0x55a036842dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- 172.21.15.63:0/2761441431 shutdown_connections 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- 172.21.15.63:0/2761441431 wait complete. 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.691+0000 7fcff087c740 1 -- start start 2024-04-25T18:58:37.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 Processor -- start 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03683f9b0 0x55a036852eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a036850730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b2730 con 0x55a03684ab90 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b3ba0 con 0x55a03683f9b0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81b2e00 con 0x55a0368402d0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.692+0000 7fcff087c740 10 client.0 _ll_register_callbacks cb 0x7fcfd0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33088/0 (socket says 172.21.15.63:33088) 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 -- 172.21.15.63:0/3564472148 learned_addr learned my addr 172.21.15.63:0/3564472148 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03683f9b0 0x55a036852eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a036850730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 -- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03683f9b0 msgr2=0x55a036852eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03683f9b0 0x55a036852eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 -- 172.21.15.63:0/3564472148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 msgr2=0x55a036850730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a036850730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 -- 172.21.15.63:0/3564472148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a036850e70 con 0x55a0368402d0 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfde000640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009bf0 tx=0x7fcfd0030ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fe2f19a993ceca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.693+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0059070 con 0x55a0368402d0 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7fcfd002e040 con 0x55a0368402d0 2024-04-25T18:58:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfd0036040 con 0x55a0368402d0 2024-04-25T18:58:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fcfd0048040 con 0x55a0368402d0 2024-04-25T18:58:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 10 client.26626 fetch_fsmap learned FSMap version 47 2024-04-25T18:58:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a0368c1390 con 0x55a0368402d0 2024-04-25T18:58:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd0036040 con 0x55a0368402d0 2024-04-25T18:58:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7fcfd0053030 con 0x55a0368402d0 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 10 client.26626 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 20 client.26626 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a0368c18f0 0x55a0368c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 4 client.26626 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.694+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0367b3320 con 0x55a0368c18f0 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.695+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a0368c18f0 0x55a0368c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.695+0000 7fcfe7400640 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a0368c18f0 0x55a0368c3d10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b97ef80797bc3aa server_cookie=1191ecdb40631da0 in_seq=0 out_seq=0 2024-04-25T18:58:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.695+0000 7fcfd4c00640 10 client.26626 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:37.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.721+0000 7fcfd4c00640 1 -- 172.21.15.63:0/3564472148 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0367b3320 con 0x55a0368c18f0 2024-04-25T18:58:37.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.721+0000 7fcfd4c00640 10 client.26626 handle_command_reply: tid=0 2024-04-25T18:58:37.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.721+0000 7fcff087c740 1 client.26626 shutdown 2024-04-25T18:58:37.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 msgr2=0x55a0368501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.724 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009bf0 tx=0x7fcfd0030ed0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 shutdown_connections 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a0368c18f0 0x55a0368c3d10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a03684ab90 0x55a036850730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a0368402d0 0x55a0368501f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 --2- 172.21.15.63:0/3564472148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a03683f9b0 0x55a036852eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcfd4c00640 0 client.26626 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 >> 172.21.15.63:0/3564472148 conn(0x55a03683bec0 msgr2=0x55a036847a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 shutdown_connections 2024-04-25T18:58:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 1 -- 172.21.15.63:0/3564472148 wait complete. 2024-04-25T18:58:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.722+0000 7fcff087c740 20 client.26626 trim_cache size 0 max 16384 2024-04-25T18:58:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:58:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:58:37.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:58:37.727 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 msgr2=0x7fcfcc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 0x7fcfcc0a61d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55a036840f50 tx=0x7fcfd80d47a0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 msgr2=0x7fcfe81a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e9f0 tx=0x7fcfd800edf0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 shutdown_connections 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fcfcc0a3d10 0x7fcfcc0a61d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fcfe810e420 0x7fcfe819e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfe8108e10 0x7fcfe81a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 --2- 172.21.15.63:0/3069268600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe8108490 0x7fcfe81a4810 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 >> 172.21.15.63:0/3069268600 conn(0x7fcfe8077d30 msgr2=0x7fcfe8078ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 shutdown_connections 2024-04-25T18:58:37.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:37.723+0000 7fcfe7e00640 1 -- 172.21.15.63:0/3069268600 wait complete. 2024-04-25T18:58:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 -- 172.21.15.63:0/3727227102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c10e3b0 msgr2=0x7f7e4c115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 --2- 172.21.15.63:0/3727227102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c115f80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7e4400b0d0 tx=0x7f7e4400b3e0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 -- 172.21.15.63:0/3727227102 shutdown_connections 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 --2- 172.21.15.63:0/3727227102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c115f80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 --2- 172.21.15.63:0/3727227102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 --2- 172.21.15.63:0/3727227102 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c109c80 0x7f7e4c10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.195+0000 7f7e51000700 1 -- 172.21.15.63:0/3727227102 >> 172.21.15.63:0/3727227102 conn(0x7f7e4c076210 msgr2=0x7f7e4c107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.196+0000 7f7e51000700 1 -- 172.21.15.63:0/3727227102 shutdown_connections 2024-04-25T18:58:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.196+0000 7f7e51000700 1 -- 172.21.15.63:0/3727227102 wait complete. 2024-04-25T18:58:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 Processor -- start 2024-04-25T18:58:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 -- start start 2024-04-25T18:58:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 0x7f7e4c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c1a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1192a0 con 0x7f7e4c109c80 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c119120 con 0x7f7e4c10e3b0 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.197+0000 7f7e51000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c119420 con 0x7f7e4c1092d0 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 0x7f7e4c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 0x7f7e4c19fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42980/0 (socket says 172.21.15.63:42980) 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49400700 1 -- 172.21.15.63:0/2600784858 learned_addr learned my addr 172.21.15.63:0/2600784858 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e4a800700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c1a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 -- 172.21.15.63:0/2600784858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c10e3b0 msgr2=0x7f7e4c1a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c1a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 -- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 msgr2=0x7f7e4c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 0x7f7e4c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.198+0000 7f7e49e00700 1 -- 172.21.15.63:0/2600784858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c1a4730 con 0x7f7e4c1092d0 2024-04-25T18:58:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e49e00700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c19f770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e9f0 tx=0x7f7e3c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00cb70 con 0x7f7e4c1092d0 2024-04-25T18:58:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4c1a4a20 con 0x7f7e4c1092d0 2024-04-25T18:58:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e3c00ee90 con 0x7f7e4c1092d0 2024-04-25T18:58:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c003d40 con 0x7f7e4c1092d0 2024-04-25T18:58:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.199+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4c1ac2a0 con 0x7f7e4c1092d0 2024-04-25T18:58:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.200+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e4c069730 con 0x7f7e4c1092d0 2024-04-25T18:58:38.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.202+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7e3c003ee0 con 0x7f7e4c1092d0 2024-04-25T18:58:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.204+0000 7f7e42a00700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 0x7f7e300a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:38.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.204+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f7e3c0d09a0 con 0x7f7e4c1092d0 2024-04-25T18:58:38.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.204+0000 7f7e49400700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 0x7f7e300a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:38.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.205+0000 7f7e49400700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 0x7f7e300a6270 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009680 tx=0x7f7e38009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:38.210 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.208+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e3c09cfa0 con 0x7f7e4c1092d0 2024-04-25T18:58:38.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.490+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e4c1a0a80 con 0x7f7e4c1092d0 2024-04-25T18:58:38.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.492+0000 7f7e42a00700 1 -- 172.21.15.63:0/2600784858 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+11875 (secure 0 0 0) 0x7f7e3c09c8a0 con 0x7f7e4c1092d0 2024-04-25T18:58:38.493 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:58:38.494 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:58:29.003855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 msgr2=0x7f7e300a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 0x7f7e300a6270 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009680 tx=0x7f7e38009260 comp rx=0 tx=0).stop 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 msgr2=0x7f7e4c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c19f770 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c00e9f0 tx=0x7f7e3c00ed00 comp rx=0 tx=0).stop 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 shutdown_connections 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7e300a3db0 0x7f7e300a6270 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7e4c1092d0 0x7f7e4c19f770 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7e4c10e3b0 0x7f7e4c1a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 --2- 172.21.15.63:0/2600784858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e4c109c80 0x7f7e4c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 >> 172.21.15.63:0/2600784858 conn(0x7f7e4c076210 msgr2=0x7f7e4c0767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 shutdown_connections 2024-04-25T18:58:38.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:38.495+0000 7f7e51000700 1 -- 172.21.15.63:0/2600784858 wait complete. 2024-04-25T18:58:38.498 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 47 2024-04-25T18:58:38.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.25 MB/sec execute 517 sec latency 25937.666 ms 2024-04-25T18:58:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:38 smithi063 ceph-mon[28006]: pgmap v379: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 597 KiB/s wr, 36 op/s 2024-04-25T18:58:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:38 smithi070 ceph-mon[32820]: pgmap v379: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 597 KiB/s wr, 36 op/s 2024-04-25T18:58:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:38 smithi114 ceph-mon[32988]: pgmap v379: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 597 KiB/s wr, 36 op/s 2024-04-25T18:58:39.418 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id eef41991 --release --await' 2024-04-25T18:58:39.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id eef41991 --release --await 2024-04-25T18:58:39.491 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:39 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2600784858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:39.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.24 MB/sec execute 518 sec latency 26937.761 ms 2024-04-25T18:58:39.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:39 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2600784858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:39.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:39 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2600784858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:39.886 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:40.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.24 MB/sec execute 519 sec latency 27937.831 ms 2024-04-25T18:58:40.620 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:40 smithi063 ceph-mon[28006]: pgmap v380: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 880 KiB/s wr, 57 op/s 2024-04-25T18:58:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:40 smithi070 ceph-mon[32820]: pgmap v380: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 880 KiB/s wr, 57 op/s 2024-04-25T18:58:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:40 smithi114 ceph-mon[32988]: pgmap v380: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 880 KiB/s wr, 57 op/s 2024-04-25T18:58:41.516 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376206 3.23 MB/sec execute 520 sec latency 28937.926 ms 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 -- 172.21.15.63:0/2979376533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc1034b0 msgr2=0x7f5cfc10f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 --2- 172.21.15.63:0/2979376533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc10f970 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009990 tx=0x7f5cf4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 -- 172.21.15.63:0/2979376533 shutdown_connections 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 --2- 172.21.15.63:0/2979376533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc10f970 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 --2- 172.21.15.63:0/2979376533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 0x7f5cfc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 --2- 172.21.15.63:0/2979376533 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc102af0 0x7f5cfc102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 -- 172.21.15.63:0/2979376533 >> 172.21.15.63:0/2979376533 conn(0x7f5cfc079b90 msgr2=0x7f5cfc079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 -- 172.21.15.63:0/2979376533 shutdown_connections 2024-04-25T18:58:42.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.304+0000 7f5d02a00700 1 -- 172.21.15.63:0/2979376533 wait complete. 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.305+0000 7f5d02a00700 1 Processor -- start 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.305+0000 7f5d02a00700 1 -- start start 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 0x7f5cfc0757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc112cf0 con 0x7f5cfc102af0 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc112b70 con 0x7f5cfc108aa0 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d02a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc112e70 con 0x7f5cfc1034b0 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42998/0 (socket says 172.21.15.63:42998) 2024-04-25T18:58:42.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5cfbe00700 1 -- 172.21.15.63:0/1292211414 learned_addr learned my addr 172.21.15.63:0/1292211414 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.306+0000 7f5d00a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 0x7f5cfc0757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 -- 172.21.15.63:0/1292211414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 msgr2=0x7f5cfc0757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 0x7f5cfc0757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 -- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 msgr2=0x7f5cfc074d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 -- 172.21.15.63:0/1292211414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc1a7680 con 0x7f5cfc1034b0 2024-04-25T18:58:42.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf3400700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc075280 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e9f0 tx=0x7f5ce400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf9600700 1 -- 172.21.15.63:0/1292211414 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400c9c0 con 0x7f5cfc1034b0 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc1a7970 con 0x7f5cfc1034b0 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf9600700 1 -- 172.21.15.63:0/1292211414 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce400cb60 con 0x7f5cfc1034b0 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.307+0000 7f5cf9600700 1 -- 172.21.15.63:0/1292211414 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4018680 con 0x7f5cfc1034b0 2024-04-25T18:58:42.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.308+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc1a7eb0 con 0x7f5cfc1034b0 2024-04-25T18:58:42.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.313+0000 7f5cf9600700 1 -- 172.21.15.63:0/1292211414 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5ce4010430 con 0x7f5cfc1034b0 2024-04-25T18:58:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 Processor -- start 2024-04-25T18:58:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 -- start start 2024-04-25T18:58:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5cf9600700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 0x7f5ce00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5cf9600700 1 -- 172.21.15.63:0/1292211414 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f5ce40d09c0 con 0x7f5cfc1034b0 2024-04-25T18:58:42.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a91d34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c7a9118f90 con 0x55c7a91c5920 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c7a90bf830 con 0x55c7a91d10f0 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.314+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c7a9114f50 con 0x55c7a91c6240 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5d00a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 0x7f5ce00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:33138/0 (socket says 172.21.15.63:33138) 2024-04-25T18:58:42.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 -- 172.21.15.63:0/810951657 learned_addr learned my addr 172.21.15.63:0/810951657 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5d00a00700 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a91d34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 -- 172.21.15.63:0/810951657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 msgr2=0x55c7a91d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a91d34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 -- 172.21.15.63:0/810951657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 msgr2=0x55c7a91c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cf3400700 1 -- 172.21.15.63:0/810951657 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a90b5780 con 0x55c7a91c6240 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.315+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.316+0000 7f5cf3400700 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40a5000 tx=0x7f5ce409c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea779aba7ac3ec0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.316+0000 7f5cf1600700 1 -- 172.21.15.63:0/810951657 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400cd10 con 0x55c7a91c6240 2024-04-25T18:58:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.316+0000 7f5d00a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 0x7f5ce00a6270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4006010 tx=0x7f5cf4009ca0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.316+0000 7f5cf1600700 1 -- 172.21.15.63:0/810951657 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce409abc0 con 0x55c7a91c6240 2024-04-25T18:58:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.316+0000 7f5cf1600700 1 -- 172.21.15.63:0/810951657 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40a50a0 con 0x55c7a91c6240 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/810951657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 msgr2=0x55c7a91d0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40a5000 tx=0x7f5ce409c530 comp rx=0 tx=0).stop 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/810951657 shutdown_connections 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a91d34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/810951657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91d0bb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/810951657 >> 172.21.15.63:0/810951657 conn(0x55c7a90f46a0 msgr2=0x55c7a91c1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/810951657 shutdown_connections 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.317+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/810951657 wait complete. 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.318+0000 7f5d0d2acb80 1 -- start start 2024-04-25T18:58:42.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.318+0000 7f5d0d2acb80 1 Processor -- start 2024-04-25T18:58:42.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91d10f0 0x55c7a91ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c7a91d3c10 con 0x55c7a91d10f0 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c7a90d2960 con 0x55c7a91c5920 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c7a91d3d90 con 0x55c7a91c6240 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d0d2acb80 10 client.0 _ll_register_callbacks cb 0x55c7a90b1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56460/0 (socket says 172.21.15.63:56460) 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cfbe00700 1 -- 172.21.15.63:0/2252145429 learned_addr learned my addr 172.21.15.63:0/2252145429 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5d00a00700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91d10f0 0x55c7a91ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cf3400700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cf3400700 1 -- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 msgr2=0x55c7a91d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cf3400700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.319+0000 7f5cf3400700 1 -- 172.21.15.63:0/2252145429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91d10f0 msgr2=0x55c7a91ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cf3400700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91d10f0 0x55c7a91ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cf3400700 1 -- 172.21.15.63:0/2252145429 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a90669d0 con 0x55c7a91c6240 2024-04-25T18:58:42.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cf3400700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91cdbb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x55c7a91c67c0 tx=0x7f5ce40ce010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db0f1e310ed53e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cea000700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce409cd30 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce409abc0 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.320+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f5ce40a50a0 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5ce40a52a0 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 10 client.26644 fetch_fsmap learned FSMap version 47 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55c7a90afbf0 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce409abc0 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f5ce4098610 con 0x55c7a91c6240 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 10 client.26644 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 10 client.26644 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 20 client.26644 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a91e0810 0x55c7a91e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 4 client.26644 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:42.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.321+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7a9126780 con 0x55c7a91e0810 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.322+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a91e0810 0x55c7a91e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.322+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a91e0810 0x55c7a91e2c00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd35702929dac9d5 server_cookie=8e603e76e7cb50fc in_seq=0 out_seq=0 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.322+0000 7f5cea000700 10 client.26644 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5cea000700 1 -- 172.21.15.63:0/2252145429 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7a9126780 con 0x55c7a91e0810 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5cea000700 10 client.26644 handle_command_reply: tid=0 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 client.26644 shutdown 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 msgr2=0x55c7a91cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91cdbb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x55c7a91c67c0 tx=0x7f5ce40ce010 comp rx=0 tx=0).stop 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 shutdown_connections 2024-04-25T18:58:42.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a91e0810 0x55c7a91e2c00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91d10f0 0x55c7a91ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c5920 0x55c7a91d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5cea000700 0 client.26644 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/2252145429 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a91cdbb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 >> 172.21.15.63:0/2252145429 conn(0x55c7a90f46a0 msgr2=0x55c7a91c19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.323+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 shutdown_connections 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.324+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/2252145429 wait complete. 2024-04-25T18:58:42.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.324+0000 7f5d0d2acb80 20 client.26644 trim_cache size 0 max 16384 2024-04-25T18:58:42.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 Processor -- start 2024-04-25T18:58:42.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 -- start start 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91d10f0 0x55c7a91dc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1b2030 con 0x55c7a91c5920 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1af830 con 0x55c7a91c6240 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.346+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ad6a0 con 0x55c7a91d10f0 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56470/0 (socket says 172.21.15.63:56470) 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 -- 172.21.15.63:0/425111117 learned_addr learned my addr 172.21.15.63:0/425111117 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5d00a00700 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91d10f0 0x55c7a91dc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 -- 172.21.15.63:0/425111117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91d10f0 msgr2=0x55c7a91dc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91d10f0 0x55c7a91dc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 -- 172.21.15.63:0/425111117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 msgr2=0x55c7a91e5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91e5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 -- 172.21.15.63:0/425111117 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a91cc120 con 0x55c7a91c6240 2024-04-25T18:58:42.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.347+0000 7f5cf3400700 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4098d20 tx=0x7f5ce4009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90c49fe8f6acd464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.349+0000 7f5cf0c00700 1 -- 172.21.15.63:0/425111117 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4067350 con 0x55c7a91c6240 2024-04-25T18:58:42.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.349+0000 7f5cf0c00700 1 -- 172.21.15.63:0/425111117 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce409cd30 con 0x55c7a91c6240 2024-04-25T18:58:42.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.349+0000 7f5cf0c00700 1 -- 172.21.15.63:0/425111117 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce409b990 con 0x55c7a91c6240 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/425111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 msgr2=0x55c7a91e6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4098d20 tx=0x7f5ce4009790 comp rx=0 tx=0).stop 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/425111117 shutdown_connections 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91d10f0 0x55c7a91dc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a91e5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/425111117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91c6240 0x55c7a91e6170 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/425111117 >> 172.21.15.63:0/425111117 conn(0x55c7a90f46a0 msgr2=0x55c7a91dc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/425111117 shutdown_connections 2024-04-25T18:58:42.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.350+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/425111117 wait complete. 2024-04-25T18:58:42.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.351+0000 7f5d0d2acb80 1 -- start start 2024-04-25T18:58:42.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.351+0000 7f5d0d2acb80 1 Processor -- start 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.351+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a9240120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a9240660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a923a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1adcf0 con 0x55c7a91c5920 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1af160 con 0x55c7a91d10f0 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1ae3c0 con 0x55c7a91c6240 2024-04-25T18:58:42.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d0d2acb80 10 client.0 _ll_register_callbacks cb 0x55c7a90b1820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:42.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d00a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a923a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a9240120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a9240660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d00a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a923a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56486/0 (socket says 172.21.15.63:56486) 2024-04-25T18:58:42.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5d00a00700 1 -- 172.21.15.63:0/467758285 learned_addr learned my addr 172.21.15.63:0/467758285 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:42.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 -- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 msgr2=0x55c7a923a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a923a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 -- 172.21.15.63:0/467758285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 msgr2=0x55c7a9240120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a9240120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cf3400700 1 -- 172.21.15.63:0/467758285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a923b090 con 0x55c7a91c6240 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.352+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a9240120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5cf3400700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a9240660 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4098d20 tx=0x7f5ce409c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=470a196633ebd8f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5cea000700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce409b990 con 0x55c7a91c6240 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce409cd30 con 0x55c7a91c6240 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f5ce409bd10 con 0x55c7a91c6240 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5ce409bb30 con 0x55c7a91c6240 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5d0d2acb80 10 client.26650 fetch_fsmap learned FSMap version 47 2024-04-25T18:58:42.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.353+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55c7a923b090 con 0x55c7a91c6240 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ce409cd30 con 0x55c7a91c6240 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 47) ==== 8358+0+0 (secure 0 0 0) 0x7f5ce40bba80 con 0x55c7a91c6240 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 10 client.26650 fetch_fsmap finished waiting for FSMap version 47 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 10 client.26650 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 20 client.26650 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a9247b10 0x55c7a9249f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 4 client.26650 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "eef41991", "release": true, "await": true}] 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "eef41991", "release": true, "await": true}) -- 0x55c7a9120820 con 0x55c7a9247b10 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.354+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a9247b10 0x55c7a9249f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:42.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.355+0000 7f5cfbe00700 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a9247b10 0x55c7a9249f00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa4949e1da1dc7e server_cookie=1eb5408e206df5d8 in_seq=0 out_seq=0 2024-04-25T18:58:42.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.355+0000 7f5cea000700 10 client.26650 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.364+0000 7f5cea000700 1 -- 172.21.15.63:0/467758285 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55c7a9120820 con 0x55c7a9247b10 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.364+0000 7f5cea000700 10 client.26650 handle_command_reply: tid=0 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.364+0000 7f5d0d2acb80 1 client.26650 shutdown 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.364+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 msgr2=0x55c7a9240660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.364+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a9240660 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4098d20 tx=0x7f5ce409c7c0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 shutdown_connections 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c7a9247b10 0x55c7a9249f00 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c7a91d10f0 0x55c7a923a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7a91c5920 0x55c7a9240120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5cea000700 0 client.26650 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 --2- 172.21.15.63:0/467758285 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c7a91c6240 0x55c7a9240660 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 >> 172.21.15.63:0/467758285 conn(0x55c7a90f46a0 msgr2=0x55c7a91dc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 shutdown_connections 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 1 -- 172.21.15.63:0/467758285 wait complete. 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.365+0000 7f5d0d2acb80 20 client.26650 trim_cache size 0 max 16384 2024-04-25T18:58:42.367 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 47, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 36, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "eef41991": { 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 36, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:58:42.368 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 81.5, 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:42.369 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 msgr2=0x7f5ce00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 0x7f5ce00a6270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4006010 tx=0x7f5cf4009ca0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 msgr2=0x7f5cfc075280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc075280 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e9f0 tx=0x7f5ce400ed00 comp rx=0 tx=0).stop 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 shutdown_connections 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5ce00a3db0 0x7f5ce00a6270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cfc108aa0 0x7f5cfc0757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5cfc102af0 0x7f5cfc074d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 --2- 172.21.15.63:0/1292211414 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5cfc1034b0 0x7f5cfc075280 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:42.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 >> 172.21.15.63:0/1292211414 conn(0x7f5cfc079b90 msgr2=0x7f5cfc100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:42.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 shutdown_connections 2024-04-25T18:58:42.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:42.366+0000 7f5d02a00700 1 -- 172.21.15.63:0/1292211414 wait complete. 2024-04-25T18:58:42.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376246 3.23 MB/sec execute 521 sec latency 29794.610 ms 2024-04-25T18:58:42.619 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:42 smithi114 ceph-mon[32988]: pgmap v381: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:42.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:42 smithi063 ceph-mon[28006]: pgmap v381: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:42 smithi070 ceph-mon[32820]: pgmap v381: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:43.366 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'eef41991', seconds elapsed: 38.4 2024-04-25T18:58:43.366 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.5 2024-04-25T18:58:43.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:58:43.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 376801 3.22 MB/sec execute 522 sec latency 20.109 ms 2024-04-25T18:58:43.868 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:44.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 377666 3.22 MB/sec execute 523 sec latency 164.768 ms 2024-04-25T18:58:44.537 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:44 smithi063 ceph-mon[28006]: pgmap v382: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:44 smithi070 ceph-mon[32820]: pgmap v382: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:44 smithi114 ceph-mon[32988]: pgmap v382: 97 pgs: 97 active+clean; 111 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 541 KiB/s wr, 35 op/s 2024-04-25T18:58:45.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 377666 3.22 MB/sec execute 524 sec latency 1165.350 ms 2024-04-25T18:58:45.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:45 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:45.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:45 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:45.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:45 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 -- 172.21.15.63:0/549452730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd84109c80 msgr2=0x7efd8410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 --2- 172.21.15.63:0/549452730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd84109c80 0x7efd8410dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7efd70009990 tx=0x7efd70009ca0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 -- 172.21.15.63:0/549452730 shutdown_connections 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 --2- 172.21.15.63:0/549452730 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd8410e3b0 0x7efd84115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 --2- 172.21.15.63:0/549452730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd841092d0 0x7efd841096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 --2- 172.21.15.63:0/549452730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd84109c80 0x7efd8410dcd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 -- 172.21.15.63:0/549452730 >> 172.21.15.63:0/549452730 conn(0x7efd84076210 msgr2=0x7efd84078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 -- 172.21.15.63:0/549452730 shutdown_connections 2024-04-25T18:58:46.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.217+0000 7efd8b800700 1 -- 172.21.15.63:0/549452730 wait complete. 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.218+0000 7efd8b800700 1 Processor -- start 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 -- start start 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd84109c80 0x7efd841a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 0x7efd8419f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:46.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efd84119270 con 0x7efd8410e3b0 2024-04-25T18:58:46.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd841190f0 con 0x7efd841092d0 2024-04-25T18:58:46.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd8b800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efd841193f0 con 0x7efd84109c80 2024-04-25T18:58:46.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd88e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd88e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56502/0 (socket says 172.21.15.63:56502) 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.219+0000 7efd88e00700 1 -- 172.21.15.63:0/3230064717 learned_addr learned my addr 172.21.15.63:0/3230064717 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd89800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 0x7efd8419f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd83e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd84109c80 0x7efd841a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 -- 172.21.15.63:0/3230064717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd84109c80 msgr2=0x7efd841a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd84109c80 0x7efd841a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 -- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 msgr2=0x7efd8419f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 0x7efd8419f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 -- 172.21.15.63:0/3230064717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8419ff70 con 0x7efd841092d0 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd89800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 0x7efd8419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:58:46.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.220+0000 7efd88e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd7800ef10 tx=0x7efd7800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:46.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.221+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7800ca70 con 0x7efd841092d0 2024-04-25T18:58:46.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.221+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd78019070 con 0x7efd841092d0 2024-04-25T18:58:46.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.221+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78014480 con 0x7efd841092d0 2024-04-25T18:58:46.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.221+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd841a0200 con 0x7efd841092d0 2024-04-25T18:58:46.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.221+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd841ac4f0 con 0x7efd841092d0 2024-04-25T18:58:46.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.222+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8410a7b0 con 0x7efd841092d0 2024-04-25T18:58:46.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.224+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efd78012050 con 0x7efd841092d0 2024-04-25T18:58:46.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.226+0000 7efd81600700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 0x7efd6c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:46.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.226+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7efd780d0b40 con 0x7efd841092d0 2024-04-25T18:58:46.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.226+0000 7efd83e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 0x7efd6c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:46.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.227+0000 7efd83e00700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 0x7efd6c0a62f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd841a0e90 tx=0x7efd70005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:46.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.228+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd78010040 con 0x7efd841092d0 2024-04-25T18:58:46.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 378505 3.22 MB/sec execute 525 sec latency 1547.260 ms 2024-04-25T18:58:46.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.572+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd841a0b70 con 0x7efd841092d0 2024-04-25T18:58:46.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.573+0000 7efd81600700 1 -- 172.21.15.63:0/3230064717 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11877 (secure 0 0 0) 0x7efd7809d140 con 0x7efd841092d0 2024-04-25T18:58:46.575 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:58:46.575 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:58:44.004081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:58:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.577+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 msgr2=0x7efd6c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.577+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 0x7efd6c0a62f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd841a0e90 tx=0x7efd70005fd0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.577+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 msgr2=0x7efd841a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.577+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7efd7800ef10 tx=0x7efd7800c3a0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 shutdown_connections 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efd6c0a3e30 0x7efd6c0a62f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efd8410e3b0 0x7efd8419f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd841092d0 0x7efd841a5570 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 --2- 172.21.15.63:0/3230064717 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efd84109c80 0x7efd841a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 >> 172.21.15.63:0/3230064717 conn(0x7efd84076210 msgr2=0x7efd84078580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.578+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 shutdown_connections 2024-04-25T18:58:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:46.579+0000 7efd8b800700 1 -- 172.21.15.63:0/3230064717 wait complete. 2024-04-25T18:58:46.581 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 48 2024-04-25T18:58:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:46 smithi070 ceph-mon[32820]: pgmap v383: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 984 KiB/s wr, 67 op/s 2024-04-25T18:58:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:46 smithi114 ceph-mon[32988]: pgmap v383: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 984 KiB/s wr, 67 op/s 2024-04-25T18:58:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:46 smithi063 ceph-mon[28006]: pgmap v383: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 984 KiB/s wr, 67 op/s 2024-04-25T18:58:47.517 INFO:tasks.workunit.client.0.smithi063.stdout: 1 379770 3.23 MB/sec execute 526 sec latency 19.150 ms 2024-04-25T18:58:47.588 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 81.5 --await-for 42.0' 2024-04-25T18:58:47.589 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 81.5 --await-for 42.0 2024-04-25T18:58:47.669 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:47 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3230064717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:47.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:47 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3230064717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:47.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:47 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3230064717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:58:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:58:48.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 380353 3.23 MB/sec execute 527 sec latency 25.230 ms 2024-04-25T18:58:48.608 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:48 smithi063 ceph-mon[28006]: pgmap v384: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 726 KiB/s wr, 53 op/s 2024-04-25T18:58:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:48 smithi070 ceph-mon[32820]: pgmap v384: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 726 KiB/s wr, 53 op/s 2024-04-25T18:58:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:48 smithi114 ceph-mon[32988]: pgmap v384: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 726 KiB/s wr, 53 op/s 2024-04-25T18:58:49.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381122 3.23 MB/sec execute 528 sec latency 56.827 ms 2024-04-25T18:58:50.500 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:50 smithi063 ceph-mon[28006]: pgmap v385: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-25T18:58:50.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381255 3.22 MB/sec execute 529 sec latency 821.212 ms 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 -- 172.21.15.63:0/1415317021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 msgr2=0x7ff4ec111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 --2- 172.21.15.63:0/1415317021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec111890 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0a0 tx=0x7ff4e400b3b0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 -- 172.21.15.63:0/1415317021 shutdown_connections 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 --2- 172.21.15.63:0/1415317021 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 --2- 172.21.15.63:0/1415317021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec111890 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 --2- 172.21.15.63:0/1415317021 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 -- 172.21.15.63:0/1415317021 >> 172.21.15.63:0/1415317021 conn(0x7ff4ec079b20 msgr2=0x7ff4ec079f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 -- 172.21.15.63:0/1415317021 shutdown_connections 2024-04-25T18:58:50.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.573+0000 7ff4f3000700 1 -- 172.21.15.63:0/1415317021 wait complete. 2024-04-25T18:58:50.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.574+0000 7ff4f3000700 1 Processor -- start 2024-04-25T18:58:50.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.574+0000 7ff4f3000700 1 -- start start 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.574+0000 7ff4f3000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f3000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f3000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec19faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f3000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec114e20 con 0x7ff4ec10f4a0 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f3000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec114ca0 con 0x7ff4ec075590 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f3000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec114fa0 con 0x7ff4ec074c70 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58376/0 (socket says 172.21.15.63:58376) 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2468766457 learned_addr learned my addr 172.21.15.63:0/2468766457 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec19faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.575+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2468766457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 msgr2=0x7ff4ec19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec19b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 msgr2=0x7ff4ec19faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec19faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2468766457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ec1a0230 con 0x7ff4ec074c70 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00eb30 tx=0x7ff4dc00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4e9600700 1 -- 172.21.15.63:0/2468766457 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc00cbd0 con 0x7ff4ec074c70 2024-04-25T18:58:50.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4e9600700 1 -- 172.21.15.63:0/2468766457 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4dc0044d0 con 0x7ff4ec074c70 2024-04-25T18:58:50.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4e9600700 1 -- 172.21.15.63:0/2468766457 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc003e30 con 0x7ff4ec074c70 2024-04-25T18:58:50.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ec1a0520 con 0x7ff4ec074c70 2024-04-25T18:58:50.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.576+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ec1a7da0 con 0x7ff4ec074c70 2024-04-25T18:58:50.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.580+0000 7ff4e9600700 1 -- 172.21.15.63:0/2468766457 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ff4dc004000 con 0x7ff4ec074c70 2024-04-25T18:58:50.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.581+0000 7ff4e9600700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 0x7ff4d00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.581+0000 7ff4e9600700 1 -- 172.21.15.63:0/2468766457 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7ff4dc0d0ba0 con 0x7ff4ec074c70 2024-04-25T18:58:50.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.582+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 0x7ff4d00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.582+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 0x7ff4d00a6390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec110ff0 tx=0x7ff4d4009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.583+0000 7ff4fd8fab80 1 Processor -- start 2024-04-25T18:58:50.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.583+0000 7ff4fd8fab80 1 -- start start 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 0x55e6686eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e668631020 con 0x55e6686de6f0 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e6685d78c0 con 0x55e6686e9560 2024-04-25T18:58:50.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e66862cfe0 con 0x55e6686dddd0 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4f1000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 0x55e6686eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58392/0 (socket says 172.21.15.63:58392) 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4ebe00700 1 -- 172.21.15.63:0/3487098205 learned_addr learned my addr 172.21.15.63:0/3487098205 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.584+0000 7ff4e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 -- 172.21.15.63:0/3487098205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 msgr2=0x55e6686eb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 0x55e6686eb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 -- 172.21.15.63:0/3487098205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 msgr2=0x55e6686e9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 -- 172.21.15.63:0/3487098205 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6685cd810 con 0x55e6686dddd0 2024-04-25T18:58:50.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4f1000700 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 0x55e6686eb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0a51e0 tx=0x7ff4dc098dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8d1c9a346d5be88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4e1600700 1 -- 172.21.15.63:0/3487098205 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0042f0 con 0x55e6686dddd0 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4e1600700 1 -- 172.21.15.63:0/3487098205 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4dc09d1a0 con 0x55e6686dddd0 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.585+0000 7ff4e1600700 1 -- 172.21.15.63:0/3487098205 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc09cd00 con 0x55e6686dddd0 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/3487098205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 msgr2=0x55e6686de1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0a51e0 tx=0x7ff4dc098dc0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/3487098205 shutdown_connections 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686de1b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686e9560 0x55e6686eb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/3487098205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/3487098205 >> 172.21.15.63:0/3487098205 conn(0x55e66860d0e0 msgr2=0x55e6686da2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.586+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/3487098205 shutdown_connections 2024-04-25T18:58:50.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.587+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/3487098205 wait complete. 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.587+0000 7ff4fd8fab80 1 -- start start 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.587+0000 7ff4fd8fab80 1 Processor -- start 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686e6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e6686ec050 con 0x55e6686de6f0 2024-04-25T18:58:50.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55e6685ea9f0 con 0x55e6686dddd0 2024-04-25T18:58:50.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55e6686ec1d0 con 0x55e6686e9560 2024-04-25T18:58:50.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4fd8fab80 10 client.0 _ll_register_callbacks cb 0x55e6685c98b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57652/0 (socket says 172.21.15.63:57652) 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4e3400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40682/0 (socket says 172.21.15.63:40682) 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2963724005 learned_addr learned my addr 172.21.15.63:0/2963724005 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.588+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 -- 172.21.15.63:0/2963724005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 msgr2=0x55e6686e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686e6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 -- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 msgr2=0x55e6686e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 -- 172.21.15.63:0/2963724005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e66857ea60 con 0x55e6686de6f0 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686e6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4005c30 tx=0x7ff4d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4b4e9bc3020de9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4da000700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:58:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.589+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4004460 con 0x55e6686de6f0 2024-04-25T18:58:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4dc09d1a0 con 0x55e6686de6f0 2024-04-25T18:58:50.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7ff4d401f890 con 0x55e6686de6f0 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ff4d401fa90 con 0x55e6686de6f0 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4fd8fab80 10 client.16461 fetch_fsmap learned FSMap version 48 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e6685c7c80 con 0x55e6686de6f0 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.590+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4dc09d1a0 con 0x55e6686de6f0 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7ff4d4026020 con 0x55e6686de6f0 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 10 client.16461 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 10 client.16461 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 20 client.16461 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:50.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6686f8ed0 0x55e6686fb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 4 client.16461 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e66863e810 con 0x55e6686f8ed0 2024-04-25T18:58:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.591+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6686f8ed0 0x55e6686fb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.592+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6686f8ed0 0x55e6686fb2c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e02f7b2f19cf5a6 server_cookie=ac70ce3712247ef8 in_seq=0 out_seq=0 2024-04-25T18:58:50.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.592+0000 7ff4da000700 10 client.16461 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4da000700 1 -- 172.21.15.63:0/2963724005 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e66863e810 con 0x55e6686f8ed0 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4da000700 10 client.16461 handle_command_reply: tid=0 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 client.16461 shutdown 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 msgr2=0x55e6686e60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4005c30 tx=0x7ff4d40047c0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 shutdown_connections 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6686f8ed0 0x55e6686fb2c0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686e6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686dddd0 0x55e6686e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2963724005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686de6f0 0x55e6686e60c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 >> 172.21.15.63:0/2963724005 conn(0x55e66860d0e0 msgr2=0x55e6686d9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4da000700 0 client.16461 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 shutdown_connections 2024-04-25T18:58:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.593+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2963724005 wait complete. 2024-04-25T18:58:50.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.594+0000 7ff4fd8fab80 20 client.16461 trim_cache size 0 max 16384 2024-04-25T18:58:50.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.621+0000 7ff4fd8fab80 1 Processor -- start 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.621+0000 7ff4fd8fab80 1 -- start start 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 0x55e6686e8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686da430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1b1f60 con 0x55e6686dddd0 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1af730 con 0x55e6686de6f0 2024-04-25T18:58:50.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1ad5a0 con 0x55e6686e9560 2024-04-25T18:58:50.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 0x55e6686e8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4ebe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 0x55e6686e8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40692/0 (socket says 172.21.15.63:40692) 2024-04-25T18:58:50.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4f1000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4ebe00700 1 -- 172.21.15.63:0/2542553263 learned_addr learned my addr 172.21.15.63:0/2542553263 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.622+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686da430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 -- 172.21.15.63:0/2542553263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 msgr2=0x55e6686f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686f4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 -- 172.21.15.63:0/2542553263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 msgr2=0x55e6686e8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 0x55e6686e8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 -- 172.21.15.63:0/2542553263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6686dadb0 con 0x55e6686de6f0 2024-04-25T18:58:50.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686da430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009fd0 tx=0x7ff4d40058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92dd8cf4d32e8003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.623+0000 7ff4e0c00700 1 -- 172.21.15.63:0/2542553263 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d403d070 con 0x55e6686de6f0 2024-04-25T18:58:50.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.624+0000 7ff4e0c00700 1 -- 172.21.15.63:0/2542553263 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4d4003680 con 0x55e6686de6f0 2024-04-25T18:58:50.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.624+0000 7ff4e0c00700 1 -- 172.21.15.63:0/2542553263 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40287f0 con 0x55e6686de6f0 2024-04-25T18:58:50.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.624+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2542553263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 msgr2=0x55e6686da430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.624+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686da430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4009fd0 tx=0x7ff4d40058e0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2542553263 shutdown_connections 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686e9560 0x55e6686f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686dddd0 0x55e6686e8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2542553263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686da430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2542553263 >> 172.21.15.63:0/2542553263 conn(0x55e66860d0e0 msgr2=0x55e6686d9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2542553263 shutdown_connections 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2542553263 wait complete. 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 -- start start 2024-04-25T18:58:50.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.625+0000 7ff4fd8fab80 1 Processor -- start 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686f0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686f0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1adbf0 con 0x55e6686e9560 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1af060 con 0x55e6686de6f0 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ec1ae2c0 con 0x55e6686dddd0 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4fd8fab80 10 client.0 _ll_register_callbacks cb 0x55e6685c98b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4f1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4e3400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4f1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40694/0 (socket says 172.21.15.63:40694) 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4f1000700 1 -- 172.21.15.63:0/2572493854 learned_addr learned my addr 172.21.15.63:0/2572493854 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.626+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686f0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 -- 172.21.15.63:0/2572493854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 msgr2=0x55e6686f0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686f0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 -- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 msgr2=0x55e6686f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 -- 172.21.15.63:0/2572493854 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e668767e60 con 0x55e6686e9560 2024-04-25T18:58:50.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4e3400700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4f1000700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0a0 tx=0x7ff4e4003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac47f69c2001db00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.627+0000 7ff4da000700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:58:50.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4007770 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4d4003680 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7ff4e4007910 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ff4e4033050 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4fd8fab80 10 client.16467 fetch_fsmap learned FSMap version 48 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55e668769190 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.628+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4d4003680 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7ff4e402e030 con 0x55e6686e9560 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 10 client.16467 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 10 client.16467 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 20 client.16467 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6687698e0 0x55e66876bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 4 client.16467 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.5, "await_for": 42.0}] 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 81.5, "await_for": 42.0}) -- 0x55e6686388b0 con 0x55e6687698e0 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6687698e0 0x55e66876bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4ebe00700 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6687698e0 0x55e66876bd00 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a16082170882ca7 server_cookie=44a780d08f573aca in_seq=0 out_seq=0 2024-04-25T18:58:50.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.629+0000 7ff4da000700 10 client.16467 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4da000700 1 -- 172.21.15.63:0/2572493854 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x55e6686388b0 con 0x55e6687698e0 2024-04-25T18:58:50.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4da000700 10 client.16467 handle_command_reply: tid=0 2024-04-25T18:58:50.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 client.16467 shutdown 2024-04-25T18:58:50.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 msgr2=0x55e6686f0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0a0 tx=0x7ff4e4003f90 comp rx=0 tx=0).stop 2024-04-25T18:58:50.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 shutdown_connections 2024-04-25T18:58:50.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55e6687698e0 0x55e66876bd00 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55e6686dddd0 0x55e6686f0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e6686e9560 0x55e6686f0ac0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 --2- 172.21.15.63:0/2572493854 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55e6686de6f0 0x55e6686f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4da000700 0 client.16467 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 >> 172.21.15.63:0/2572493854 conn(0x55e66860d0e0 msgr2=0x55e6686f3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.690+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 shutdown_connections 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.691+0000 7ff4fd8fab80 1 -- 172.21.15.63:0/2572493854 wait complete. 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.691+0000 7ff4fd8fab80 20 client.16467 trim_cache size 0 max 16384 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 48, 2024-04-25T18:58:50.693 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 38, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "446cb4ea": { 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 38, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 81.5, 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:58:50.694 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:58:50.695 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 msgr2=0x7ff4d00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 0x7ff4d00a6390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec110ff0 tx=0x7ff4d4009260 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 msgr2=0x7ff4ec19b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc00eb30 tx=0x7ff4dc00ee40 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 shutdown_connections 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ff4d00a3ed0 0x7ff4d00a6390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff4ec074c70 0x7ff4ec19b270 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4ec10f4a0 0x7ff4ec19faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 --2- 172.21.15.63:0/2468766457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4ec075590 0x7ff4ec19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.692+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 >> 172.21.15.63:0/2468766457 conn(0x7ff4ec079b20 msgr2=0x7ff4ec0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.693+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 shutdown_connections 2024-04-25T18:58:50.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:58:50.693+0000 7ff4f3000700 1 -- 172.21.15.63:0/2468766457 wait complete. 2024-04-25T18:58:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:50 smithi070 ceph-mon[32820]: pgmap v385: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-25T18:58:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:50 smithi114 ceph-mon[32988]: pgmap v385: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 105 op/s 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.859+0000 7f3fb3200640 1 -- 172.21.15.63:0/2319168849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 msgr2=0x7f3fb41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.859+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2319168849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41023a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009900 tx=0x7f3fa402a3f0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 -- 172.21.15.63:0/2319168849 shutdown_connections 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2319168849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4102970 0x7f3fb41069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2319168849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb4107090 0x7f3fb410ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2319168849 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41023a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 -- 172.21.15.63:0/2319168849 >> 172.21.15.63:0/2319168849 conn(0x7f3fb40fd8a0 msgr2=0x7f3fb40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 -- 172.21.15.63:0/2319168849 shutdown_connections 2024-04-25T18:58:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.860+0000 7f3fb3200640 1 -- 172.21.15.63:0/2319168849 wait complete. 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 Processor -- start 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 -- start start 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4107090 0x7f3fb41a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 0x7f3fb419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4111fd0 con 0x7f3fb419a670 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4111e50 con 0x7f3fb4107090 2024-04-25T18:58:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.861+0000 7f3fb3200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb4112150 con 0x7f3fb4101fc0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58430/0 (socket says 172.21.15.63:58430) 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2243518198 learned_addr learned my addr 172.21.15.63:0/2243518198 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb2800640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 0x7f3fb419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1400640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4107090 0x7f3fb41a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2243518198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4107090 msgr2=0x7f3fb41a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4107090 0x7f3fb41a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 msgr2=0x7f3fb419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 0x7f3fb419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2243518198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb419b270 con 0x7f3fb4101fc0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb2800640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 0x7f3fb419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.862+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4005940 tx=0x7f3fa40040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.863+0000 7f3faaa00640 1 -- 172.21.15.63:0/2243518198 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa403f030 con 0x7f3fb4101fc0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.863+0000 7f3faaa00640 1 -- 172.21.15.63:0/2243518198 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa40044c0 con 0x7f3fb4101fc0 2024-04-25T18:58:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.863+0000 7f3faaa00640 1 -- 172.21.15.63:0/2243518198 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4035070 con 0x7f3fb4101fc0 2024-04-25T18:58:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.863+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb419b500 con 0x7f3fb4101fc0 2024-04-25T18:58:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.863+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb41a7550 con 0x7f3fb4101fc0 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.866+0000 7f3fbc1c1740 1 Processor -- start 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.866+0000 7f3faaa00640 1 -- 172.21.15.63:0/2243518198 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3fa4042030 con 0x7f3fb4101fc0 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.866+0000 7f3fbc1c1740 1 -- start start 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.866+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f663b90 0x55968f665f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55968f5be900 con 0x55968f663b90 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55968f564ef0 con 0x55968f6589b0 2024-04-25T18:58:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55968f5ba8c0 con 0x55968f6592d0 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f663b90 0x55968f665f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58446/0 (socket says 172.21.15.63:58446) 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57688/0 (socket says 172.21.15.63:57688) 2024-04-25T18:58:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2810798036 learned_addr learned my addr 172.21.15.63:0/2810798036 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2810798036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 msgr2=0x55968f658d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2810798036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f663b90 msgr2=0x55968f665f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f663b90 0x55968f665f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb1e00640 1 -- 172.21.15.63:0/2810798036 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55968f2d1210 con 0x55968f6592d0 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.867+0000 7f3fb2800640 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40ee110 tx=0x7f3fa40f4d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e59d71fff0dcb1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3faaa00640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 0x7f3f940a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3fa3400640 1 -- 172.21.15.63:0/2810798036 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40f6050 con 0x55968f6592d0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3fa3400640 1 -- 172.21.15.63:0/2810798036 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa4102070 con 0x55968f6592d0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3fa3400640 1 -- 172.21.15.63:0/2810798036 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40bc430 con 0x55968f6592d0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3faaa00640 1 -- 172.21.15.63:0/2243518198 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f3fa40f0b20 con 0x7f3fb4101fc0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.868+0000 7f3fb2800640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 0x7f3f940a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/2810798036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 msgr2=0x55968f663650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40ee110 tx=0x7f3fa40f4d50 comp rx=0 tx=0).stop 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fb2800640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 0x7f3f940a6250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009fd0 tx=0x7f3fac005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/2810798036 shutdown_connections 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f663b90 0x55968f665f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f663650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/2810798036 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f6589b0 0x55968f658d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/2810798036 >> 172.21.15.63:0/2810798036 conn(0x55968f654ec0 msgr2=0x55968f6552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/2810798036 shutdown_connections 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.869+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/2810798036 wait complete. 2024-04-25T18:58:50.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 -- start start 2024-04-25T18:58:50.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 Processor -- start 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f660f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f6614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55968f666580 con 0x55968f6589b0 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55968f578310 con 0x55968f663b90 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.870+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55968f666700 con 0x55968f6592d0 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fbc1c1740 10 client.0 _ll_register_callbacks cb 0x55968f546cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:50.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f660f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40736/0 (socket says 172.21.15.63:40736) 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f660f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58448/0 (socket says 172.21.15.63:58448) 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 -- 172.21.15.63:0/82618193 learned_addr learned my addr 172.21.15.63:0/82618193 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb2800640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f6614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 -- 172.21.15.63:0/82618193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 msgr2=0x55968f660f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f660f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 -- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 msgr2=0x55968f6614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f6614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 -- 172.21.15.63:0/82618193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968f39d620 con 0x55968f6589b0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40bca60 tx=0x7f3fa4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee81800eb5fa3679 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.871+0000 7f3f9be00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4047c90 con 0x55968f6589b0 2024-04-25T18:58:50.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f3fa40f4770 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fa4102070 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3fa4049030 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 10 client.16479 fetch_fsmap learned FSMap version 48 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55968f666580 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fa4102070 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f3fa40fc070 con 0x55968f6589b0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 10 client.16479 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 10 client.16479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 20 client.16479 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6738f0 0x55968f675d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 4 client.16479 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.872+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55968f5d28b0 con 0x55968f6738f0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.873+0000 7f3fb1400640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6738f0 0x55968f675d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.873+0000 7f3fb1400640 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6738f0 0x55968f675d40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b988f2c635d48c62 server_cookie=11040b60fcfe4025 in_seq=0 out_seq=0 2024-04-25T18:58:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.873+0000 7f3f9be00640 10 client.16479 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3f9be00640 1 -- 172.21.15.63:0/82618193 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55968f5d28b0 con 0x55968f6738f0 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3f9be00640 10 client.16479 handle_command_reply: tid=0 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 client.16479 shutdown 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 msgr2=0x55968f660a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40bca60 tx=0x7f3fa4009680 comp rx=0 tx=0).stop 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 shutdown_connections 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6738f0 0x55968f675d40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f6614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f660f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/82618193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f660a20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3f9be00640 0 client.16479 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.874+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 >> 172.21.15.63:0/82618193 conn(0x55968f654ec0 msgr2=0x55968f65abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.875+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 shutdown_connections 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.875+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/82618193 wait complete. 2024-04-25T18:58:50.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.875+0000 7f3fbc1c1740 20 client.16479 trim_cache size 0 max 16384 2024-04-25T18:58:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 Processor -- start 2024-04-25T18:58:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 -- start start 2024-04-25T18:58:50.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f65b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f65f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f67b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41b26f0 con 0x55968f6589b0 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41aff00 con 0x55968f663b90 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41add70 con 0x55968f6592d0 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f65b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fb1400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f65f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f65b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40744/0 (socket says 172.21.15.63:40744) 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fb1e00640 1 -- 172.21.15.63:0/814027184 learned_addr learned my addr 172.21.15.63:0/814027184 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.892+0000 7f3fb2800640 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f67b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 -- 172.21.15.63:0/814027184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 msgr2=0x55968f67b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f67b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 -- 172.21.15.63:0/814027184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 msgr2=0x55968f65b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f65b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 -- 172.21.15.63:0/814027184 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55968f6547a0 con 0x55968f6592d0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fb1400640 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f65f8d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55968f6562e0 tx=0x7f3f9c009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=111d03da4b5c3fde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fa2a00640 1 -- 172.21.15.63:0/814027184 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c024070 con 0x55968f6592d0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fa2a00640 1 -- 172.21.15.63:0/814027184 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c01c830 con 0x55968f6592d0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.893+0000 7f3fa2a00640 1 -- 172.21.15.63:0/814027184 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c0204e0 con 0x55968f6592d0 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/814027184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 msgr2=0x55968f65f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f65f8d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55968f6562e0 tx=0x7f3f9c009500 comp rx=0 tx=0).stop 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/814027184 shutdown_connections 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f67b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6592d0 0x55968f65f8d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/814027184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6589b0 0x55968f65b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/814027184 >> 172.21.15.63:0/814027184 conn(0x55968f654ec0 msgr2=0x55968f65f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/814027184 shutdown_connections 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.894+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/814027184 wait complete. 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 -- start start 2024-04-25T18:58:50.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 Processor -- start 2024-04-25T18:58:50.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6592d0 0x55968f6d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.899 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f668bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41ae3c0 con 0x55968f6592d0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41af830 con 0x55968f663b90 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb41aea90 con 0x55968f6589b0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fbc1c1740 10 client.0 _ll_register_callbacks cb 0x7f3fa40f4b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fb1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58470/0 (socket says 172.21.15.63:58470) 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fb1e00640 1 -- 172.21.15.63:0/1806866330 learned_addr learned my addr 172.21.15.63:0/1806866330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.895+0000 7f3fb1400640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6592d0 0x55968f6d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb2800640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f668bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 -- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 msgr2=0x55968f668bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f668bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 -- 172.21.15.63:0/1806866330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6592d0 msgr2=0x55968f6d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6592d0 0x55968f6d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 -- 172.21.15.63:0/1806866330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55968f6692c0 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fb1e00640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40bad80 tx=0x7f3fa40dd110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f68552f77c25d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3f9be00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40f6030 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f3fa4035210 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f9c01c830 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.896+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3fa40db050 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 10 client.26692 fetch_fsmap learned FSMap version 48 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55968f6692c0 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f9c01c830 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f3fa40fc070 con 0x55968f6589b0 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 10 client.26692 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 10 client.26692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 20 client.26692 populate_metadata read hostname 'smithi063' 2024-04-25T18:58:50.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6d9e90 0x55968f6dc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:58:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 4 client.26692 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:58:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55968f5cc320 con 0x55968f6d9e90 2024-04-25T18:58:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.897+0000 7f3fb1400640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6d9e90 0x55968f6dc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:58:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.898+0000 7f3fb1400640 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6d9e90 0x55968f6dc2b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2d4c4877214983c server_cookie=63a11aee3d5b413c in_seq=0 out_seq=0 2024-04-25T18:58:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.898+0000 7f3f9be00640 10 client.26692 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3f9be00640 1 -- 172.21.15.63:0/1806866330 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55968f5cc320 con 0x55968f6d9e90 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3f9be00640 10 client.26692 handle_command_reply: tid=0 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 client.26692 shutdown 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 msgr2=0x55968f6d2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40bad80 tx=0x7f3fa40dd110 comp rx=0 tx=0).stop 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 shutdown_connections 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55968f6d9e90 0x55968f6dc2b0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55968f663b90 0x55968f668bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55968f6592d0 0x55968f6d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 --2- 172.21.15.63:0/1806866330 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55968f6589b0 0x55968f6d2270 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.925+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 >> 172.21.15.63:0/1806866330 conn(0x55968f654ec0 msgr2=0x55968f654020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.926+0000 7f3f9be00640 0 client.26692 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.926+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 shutdown_connections 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.926+0000 7f3fbc1c1740 1 -- 172.21.15.63:0/1806866330 wait complete. 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.926+0000 7f3fbc1c1740 20 client.26692 trim_cache size 0 max 16384 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:58:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 msgr2=0x7f3f940a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 0x7f3f940a6250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009fd0 tx=0x7f3fac005f90 comp rx=0 tx=0).stop 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 msgr2=0x7f3fb41a0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4005940 tx=0x7f3fa40040c0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 shutdown_connections 2024-04-25T18:58:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3f940a3d90 0x7f3f940a6250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb4107090 0x7f3fb41a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fb419a670 0x7f3fb419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 --2- 172.21.15.63:0/2243518198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3fb4101fc0 0x7f3fb41a0510 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 >> 172.21.15.63:0/2243518198 conn(0x7f3fb40fd8a0 msgr2=0x7f3fb40fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.927+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 shutdown_connections 2024-04-25T18:58:50.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:58:50.928+0000 7f3fb3200640 1 -- 172.21.15.63:0/2243518198 wait complete. 2024-04-25T18:58:51.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.22 MB/sec execute 530 sec latency 972.940 ms 2024-04-25T18:58:51.797 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '446cb4ea', quiesce took 8.4 seconds. Will release after: 21.5 2024-04-25T18:58:52.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.21 MB/sec execute 531 sec latency 1835.100 ms 2024-04-25T18:58:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:52 smithi070 ceph-mon[32820]: pgmap v386: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:52 smithi114 ceph-mon[32988]: pgmap v386: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:52 smithi063 ceph-mon[28006]: pgmap v386: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:53.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.21 MB/sec execute 532 sec latency 2835.265 ms 2024-04-25T18:58:54.518 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.20 MB/sec execute 533 sec latency 3835.370 ms 2024-04-25T18:58:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:54 smithi070 ceph-mon[32820]: pgmap v387: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:54 smithi114 ceph-mon[32988]: pgmap v387: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:54 smithi063 ceph-mon[28006]: pgmap v387: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 994 KiB/s wr, 85 op/s 2024-04-25T18:58:55.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.19 MB/sec execute 534 sec latency 4835.504 ms 2024-04-25T18:58:56.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.19 MB/sec execute 535 sec latency 5835.697 ms 2024-04-25T18:58:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:56 smithi070 ceph-mon[32820]: pgmap v388: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T18:58:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:56 smithi114 ceph-mon[32988]: pgmap v388: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T18:58:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:56 smithi063 ceph-mon[28006]: pgmap v388: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-04-25T18:58:57.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.18 MB/sec execute 536 sec latency 6835.801 ms 2024-04-25T18:58:58.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.18 MB/sec execute 537 sec latency 7835.938 ms 2024-04-25T18:58:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:58:58 smithi070 ceph-mon[32820]: pgmap v389: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 624 KiB/s wr, 76 op/s 2024-04-25T18:58:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:58:58 smithi114 ceph-mon[32988]: pgmap v389: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 624 KiB/s wr, 76 op/s 2024-04-25T18:58:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:58:58 smithi063 ceph-mon[28006]: pgmap v389: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 624 KiB/s wr, 76 op/s 2024-04-25T18:58:59.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.17 MB/sec execute 538 sec latency 8836.089 ms 2024-04-25T18:59:00.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.16 MB/sec execute 539 sec latency 9836.236 ms 2024-04-25T18:59:00.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:00 smithi070 ceph-mon[32820]: pgmap v390: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 KiB/s wr, 80 op/s 2024-04-25T18:59:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:00 smithi114 ceph-mon[32988]: pgmap v390: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 KiB/s wr, 80 op/s 2024-04-25T18:59:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:00 smithi063 ceph-mon[28006]: pgmap v390: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 634 KiB/s wr, 80 op/s 2024-04-25T18:59:01.519 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.16 MB/sec execute 540 sec latency 10836.354 ms 2024-04-25T18:59:02.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.15 MB/sec execute 541 sec latency 11836.453 ms 2024-04-25T18:59:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:02 smithi070 ceph-mon[32820]: pgmap v391: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:02 smithi114 ceph-mon[32988]: pgmap v391: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:02 smithi063 ceph-mon[28006]: pgmap v391: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:03.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.15 MB/sec execute 542 sec latency 12836.600 ms 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 -- 172.21.15.63:0/661146343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074c40 msgr2=0x7f1fb810e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 --2- 172.21.15.63:0/661146343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074c40 0x7f1fb810e0d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009ca0 tx=0x7f1fa002a950 comp rx=0 tx=0).stop 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 -- 172.21.15.63:0/661146343 shutdown_connections 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 --2- 172.21.15.63:0/661146343 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb810e610 0x7f1fb8110a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 --2- 172.21.15.63:0/661146343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074c40 0x7f1fb810e0d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 --2- 172.21.15.63:0/661146343 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb8074320 0x7f1fb8074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 -- 172.21.15.63:0/661146343 >> 172.21.15.63:0/661146343 conn(0x7f1fb80fd8a0 msgr2=0x7f1fb80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 -- 172.21.15.63:0/661146343 shutdown_connections 2024-04-25T18:59:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.074+0000 7f1fbe000640 1 -- 172.21.15.63:0/661146343 wait complete. 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.075+0000 7f1fbe000640 1 Processor -- start 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 -- start start 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 0x7f1fb81a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 0x7f1fb819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb8114290 con 0x7f1fb8074320 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb8114110 con 0x7f1fb8074c40 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbe000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb8114410 con 0x7f1fb810e610 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57186/0 (socket says 172.21.15.63:57186) 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fb7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 0x7f1fb81a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbcc00640 1 -- 172.21.15.63:0/1489838061 learned_addr learned my addr 172.21.15.63:0/1489838061 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.076+0000 7f1fbd600640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 0x7f1fb819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 -- 172.21.15.63:0/1489838061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 msgr2=0x7f1fb819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 0x7f1fb819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 -- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 msgr2=0x7f1fb81a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 -- 172.21.15.63:0/1489838061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb819adf0 con 0x7f1fb8074c40 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fbd600640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 0x7f1fb819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 0x7f1fb81a09b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005e30 tx=0x7f1fa0003c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb5600640 1 -- 172.21.15.63:0/1489838061 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0034040 con 0x7f1fb8074c40 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb819b080 con 0x7f1fb8074c40 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb5600640 1 -- 172.21.15.63:0/1489838061 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa0030850 con 0x7f1fb8074c40 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.077+0000 7f1fb5600640 1 -- 172.21.15.63:0/1489838061 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0004620 con 0x7f1fb8074c40 2024-04-25T18:59:04.083 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.078+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb81a7330 con 0x7f1fb8074c40 2024-04-25T18:59:04.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.082+0000 7f1fc2574740 1 Processor -- start 2024-04-25T18:59:04.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.082+0000 7f1fc2574740 1 -- start start 2024-04-25T18:59:04.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.082+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29b6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.086 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.082+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b29c3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fc2574740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a1b291c900 con 0x55a1b29c1b90 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fc2574740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a1b28c2ef0 con 0x55a1b29b72d0 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fc2574740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a1b29188c0 con 0x55a1b29b69b0 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fb5600640 1 -- 172.21.15.63:0/1489838061 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1fa0037020 con 0x7f1fb8074c40 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34610/0 (socket says 172.21.15.63:34610) 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 -- 172.21.15.63:0/1991992498 learned_addr learned my addr 172.21.15.63:0/1991992498 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbd600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29b6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b29c3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 -- 172.21.15.63:0/1991992498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 msgr2=0x55a1b29b6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29b6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 -- 172.21.15.63:0/1991992498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 msgr2=0x55a1b29c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.087 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b29c3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbcc00640 1 -- 172.21.15.63:0/1991992498 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1b262f210 con 0x55a1b29b72d0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.083+0000 7f1fbd600640 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.084+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8011630 tx=0x7f1fa8011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ebde5dbcf9d25e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.084+0000 7f1fadc00640 1 -- 172.21.15.63:0/1991992498 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa801e070 con 0x55a1b29b72d0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.084+0000 7f1fadc00640 1 -- 172.21.15.63:0/1991992498 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa800fba0 con 0x55a1b29b72d0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.084+0000 7f1fadc00640 1 -- 172.21.15.63:0/1991992498 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8019ca0 con 0x55a1b29b72d0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.084+0000 7f1fb5600640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 0x7f1f980a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 -- 172.21.15.63:0/1991992498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 msgr2=0x55a1b29c1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8011630 tx=0x7f1fa8011af0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fb5600640 1 -- 172.21.15.63:0/1489838061 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f1fa00f1430 con 0x7f1fb8074c40 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fbd600640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 0x7f1f980a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 -- 172.21.15.63:0/1991992498 shutdown_connections 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c1650 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.088 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b29c3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 --2- 172.21.15.63:0/1991992498 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29b6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 -- 172.21.15.63:0/1991992498 >> 172.21.15.63:0/1991992498 conn(0x55a1b29b2ec0 msgr2=0x55a1b29b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 -- 172.21.15.63:0/1991992498 shutdown_connections 2024-04-25T18:59:04.089 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.085+0000 7f1fc2574740 1 -- 172.21.15.63:0/1991992498 wait complete. 2024-04-25T18:59:04.092 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 -- start start 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fbd600640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 0x7f1f980a6250 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00098c0 tx=0x7f1fb0005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 Processor -- start 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29beb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b72d0 0x55a1b29bf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a1b29c47c0 con 0x55a1b29b72d0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55a1b28d6310 con 0x55a1b29c1b90 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55a1b29c4940 con 0x55a1b29b69b0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.086+0000 7f1fc2574740 10 client.0 _ll_register_callbacks cb 0x55a1b28a4cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbd600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29beb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbd600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:34620/0 (socket says 172.21.15.63:34620) 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbd600640 1 -- 172.21.15.63:0/2135808031 learned_addr learned my addr 172.21.15.63:0/2135808031 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:04.093 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b72d0 0x55a1b29bf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 -- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 msgr2=0x55a1b29bf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 -- 172.21.15.63:0/2135808031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b72d0 msgr2=0x55a1b29bf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b72d0 0x55a1b29bf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 -- 172.21.15.63:0/2135808031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1b26fb620 con 0x55a1b29b69b0 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbd600640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29beb20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800f3c0 tx=0x7f1fa8009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=994470bcd37bb664 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.087+0000 7f1fa6000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8026020 con 0x55a1b29b69b0 2024-04-25T18:59:04.094 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa800fba0 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f1fa800fd50 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1fa8031050 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fc2574740 10 client.26704 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.088+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55a1b29d0a20 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa800fba0 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f1fa8021050 con 0x55a1b29b69b0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 10 client.26704 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 10 client.26704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 20 client.26704 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b29d1e30 0x55a1b29d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 4 client.26704 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.089+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a1b29308b0 con 0x55a1b29d1e30 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.090+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b29d1e30 0x55a1b29d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.090+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b29d1e30 0x55a1b29d4220 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ef781865ad50edc server_cookie=aedcb6e9fd695e32 in_seq=0 out_seq=0 2024-04-25T18:59:04.095 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.090+0000 7f1fa6000640 10 client.26704 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fa6000640 1 -- 172.21.15.63:0/2135808031 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a1b29308b0 con 0x55a1b29d1e30 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fa6000640 10 client.26704 handle_command_reply: tid=0 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 client.26704 shutdown 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 msgr2=0x55a1b29beb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29beb20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1fa800f3c0 tx=0x7f1fa8009f10 comp rx=0 tx=0).stop 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 shutdown_connections 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b29d1e30 0x55a1b29d4220 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29bf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b72d0 0x55a1b29bf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fa6000640 0 client.26704 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 --2- 172.21.15.63:0/2135808031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29beb20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 >> 172.21.15.63:0/2135808031 conn(0x55a1b29b2ec0 msgr2=0x55a1b29b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 shutdown_connections 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.091+0000 7f1fc2574740 1 -- 172.21.15.63:0/2135808031 wait complete. 2024-04-25T18:59:04.096 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.092+0000 7f1fc2574740 20 client.26704 trim_cache size 0 max 16384 2024-04-25T18:59:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 Processor -- start 2024-04-25T18:59:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 -- start start 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 0x55a1b29cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 0x55a1b29b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29d99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81b25e0 con 0x55a1b29b69b0 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81afdf0 con 0x55a1b29c1b90 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.108+0000 7f1fc2574740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81adc60 con 0x55a1b29b72d0 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 0x55a1b29cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fbcc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 0x55a1b29cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57208/0 (socket says 172.21.15.63:57208) 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fbcc00640 1 -- 172.21.15.63:0/4030531706 learned_addr learned my addr 172.21.15.63:0/4030531706 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 0x55a1b29b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fbd600640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29d99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 -- 172.21.15.63:0/4030531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 msgr2=0x55a1b29d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29d99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 -- 172.21.15.63:0/4030531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 msgr2=0x55a1b29cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 0x55a1b29cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fb7e00640 1 -- 172.21.15.63:0/4030531706 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1b29b7c60 con 0x55a1b29b72d0 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.109+0000 7f1fbd600640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 0x55a1b29b8a70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00eec50 tx=0x7f1fa0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96aecdf781ef9626 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fad200640 1 -- 172.21.15.63:0/4030531706 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00bcfe0 con 0x55a1b29b72d0 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fad200640 1 -- 172.21.15.63:0/4030531706 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa00bdba0 con 0x55a1b29b72d0 2024-04-25T18:59:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fad200640 1 -- 172.21.15.63:0/4030531706 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00bce00 con 0x55a1b29b72d0 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fc2574740 1 -- 172.21.15.63:0/4030531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 msgr2=0x55a1b29b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.110+0000 7f1fc2574740 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 0x55a1b29b8a70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00eec50 tx=0x7f1fa0000f40 comp rx=0 tx=0).stop 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 -- 172.21.15.63:0/4030531706 shutdown_connections 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29c1b90 0x55a1b29d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b72d0 0x55a1b29b8a70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 --2- 172.21.15.63:0/4030531706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29b69b0 0x55a1b29cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 -- 172.21.15.63:0/4030531706 >> 172.21.15.63:0/4030531706 conn(0x55a1b29b2ec0 msgr2=0x55a1b29b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 -- 172.21.15.63:0/4030531706 shutdown_connections 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 -- 172.21.15.63:0/4030531706 wait complete. 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 -- start start 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.111+0000 7f1fc2574740 1 Processor -- start 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81ae2b0 con 0x55a1b29c1b90 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81af720 con 0x55a1b29b72d0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb81ae980 con 0x55a1b29b69b0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fc2574740 10 client.0 _ll_register_callbacks cb 0x7f1fa80114d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57212/0 (socket says 172.21.15.63:57212) 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 -- 172.21.15.63:0/659289625 learned_addr learned my addr 172.21.15.63:0/659289625 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 -- 172.21.15.63:0/659289625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 msgr2=0x55a1b29c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29c7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 -- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 msgr2=0x55a1b29c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.112+0000 7f1fbd600640 1 -- 172.21.15.63:0/659289625 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1b2a317f0 con 0x55a1b29c1b90 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fb7e00640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fbd600640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0020ab0 tx=0x7f1fb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62962b26d77ed9ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fa6000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb0030060 con 0x55a1b29c1b90 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 8587+0+0 (secure 0 0 0) 0x7f1fb001eba0 con 0x55a1b29c1b90 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1fa00bdba0 con 0x55a1b29c1b90 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1fb001ede0 con 0x55a1b29c1b90 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fc2574740 10 client.16503 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.113+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55a1b2a38be0 con 0x55a1b29c1b90 2024-04-25T18:59:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa00bdba0 con 0x55a1b29c1b90 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f1fb002a070 con 0x55a1b29c1b90 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 10 client.16503 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 10 client.16503 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 20 client.16503 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b2a393c0 0x55a1b2a3b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 4 client.16503 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a1b292a320 con 0x55a1b2a393c0 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.114+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b2a393c0 0x55a1b2a3b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.115+0000 7f1fbcc00640 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b2a393c0 0x55a1b2a3b7e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7738622c50bca8c server_cookie=c416c9f857f52d4 in_seq=0 out_seq=0 2024-04-25T18:59:04.119 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.115+0000 7f1fa6000640 10 client.16503 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.139+0000 7f1fa6000640 1 -- 172.21.15.63:0/659289625 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a1b292a320 con 0x55a1b2a393c0 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.139+0000 7f1fa6000640 10 client.16503 handle_command_reply: tid=0 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 client.16503 shutdown 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 msgr2=0x55a1b2a31200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1fb0020ab0 tx=0x7f1fb00046c0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 shutdown_connections 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55a1b2a393c0 0x55a1b2a3b7e0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55a1b29b72d0 0x55a1b29c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a1b29c1b90 0x55a1b2a31200 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.142 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 --2- 172.21.15.63:0/659289625 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55a1b29b69b0 0x55a1b29c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 >> 172.21.15.63:0/659289625 conn(0x55a1b29b2ec0 msgr2=0x55a1b29b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fa6000640 0 client.16503 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 shutdown_connections 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.140+0000 7f1fc2574740 1 -- 172.21.15.63:0/659289625 wait complete. 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fc2574740 20 client.16503 trim_cache size 0 max 16384 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:59:04.143 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 msgr2=0x7f1f980a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 0x7f1f980a6250 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1fb00098c0 tx=0x7f1fb0005f90 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 msgr2=0x7f1fb81a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 0x7f1fb81a09b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0005e30 tx=0x7f1fa0003c90 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 shutdown_connections 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1f980a3d90 0x7f1f980a6250 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1fb8074c40 0x7f1fb81a09b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1fb810e610 0x7f1fb819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.141+0000 7f1fbe000640 1 --2- 172.21.15.63:0/1489838061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fb8074320 0x7f1fb81a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:04.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.142+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 >> 172.21.15.63:0/1489838061 conn(0x7f1fb80fd8a0 msgr2=0x7f1fb80fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:04.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.142+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 shutdown_connections 2024-04-25T18:59:04.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:04.142+0000 7f1fbe000640 1 -- 172.21.15.63:0/1489838061 wait complete. 2024-04-25T18:59:04.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.14 MB/sec execute 543 sec latency 13836.733 ms 2024-04-25T18:59:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:04 smithi070 ceph-mon[32820]: pgmap v392: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:04 smithi114 ceph-mon[32988]: pgmap v392: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:04 smithi063 ceph-mon[28006]: pgmap v392: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 83 KiB/s wr, 28 op/s 2024-04-25T18:59:05.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.13 MB/sec execute 544 sec latency 14836.849 ms 2024-04-25T18:59:05.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:05 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:05 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:05.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:05 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T18:59:06.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.13 MB/sec execute 545 sec latency 15836.992 ms 2024-04-25T18:59:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:06 smithi070 ceph-mon[32820]: pgmap v393: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 486 KiB/s wr, 56 op/s 2024-04-25T18:59:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:06 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:06 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:06 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:59:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:06 smithi114 ceph-mon[32988]: pgmap v393: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 486 KiB/s wr, 56 op/s 2024-04-25T18:59:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:06 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:06 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:06 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:59:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:06 smithi063 ceph-mon[28006]: pgmap v393: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 486 KiB/s wr, 56 op/s 2024-04-25T18:59:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:06 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T18:59:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:06 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T18:59:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:06 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T18:59:07.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.12 MB/sec execute 546 sec latency 16837.154 ms 2024-04-25T18:59:08.520 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.12 MB/sec execute 547 sec latency 17837.296 ms 2024-04-25T18:59:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:08 smithi114 ceph-mon[32988]: pgmap v394: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 413 KiB/s wr, 31 op/s 2024-04-25T18:59:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:08 smithi063 ceph-mon[28006]: pgmap v394: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 413 KiB/s wr, 31 op/s 2024-04-25T18:59:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:08 smithi070 ceph-mon[32820]: pgmap v394: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 413 KiB/s wr, 31 op/s 2024-04-25T18:59:09.521 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.11 MB/sec execute 548 sec latency 18837.636 ms 2024-04-25T18:59:10.521 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.11 MB/sec execute 549 sec latency 19837.804 ms 2024-04-25T18:59:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:10 smithi114 ceph-mon[32988]: pgmap v395: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 460 KiB/s wr, 47 op/s 2024-04-25T18:59:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:10 smithi114 ceph-mon[32988]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T18:59:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:10 smithi063 ceph-mon[28006]: pgmap v395: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 460 KiB/s wr, 47 op/s 2024-04-25T18:59:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:10 smithi063 ceph-mon[28006]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T18:59:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:10 smithi070 ceph-mon[32820]: pgmap v395: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 460 KiB/s wr, 47 op/s 2024-04-25T18:59:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:10 smithi070 ceph-mon[32820]: osdmap e112: 12 total, 12 up, 12 in 2024-04-25T18:59:11.521 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.10 MB/sec execute 550 sec latency 20837.971 ms 2024-04-25T18:59:12.521 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.09 MB/sec execute 551 sec latency 21838.107 ms 2024-04-25T18:59:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:12 smithi114 ceph-mon[32988]: pgmap v397: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:12 smithi063 ceph-mon[28006]: pgmap v397: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:12.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:12 smithi070 ceph-mon[32820]: pgmap v397: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:13.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T18:59:13.521 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.09 MB/sec execute 552 sec latency 22838.235 ms 2024-04-25T18:59:13.781 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:59:14.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.08 MB/sec execute 553 sec latency 23838.367 ms 2024-04-25T18:59:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:14 smithi114 ceph-mon[32988]: pgmap v398: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:14 smithi063 ceph-mon[28006]: pgmap v398: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:14 smithi070 ceph-mon[32820]: pgmap v398: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 540 KiB/s wr, 52 op/s 2024-04-25T18:59:15.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.08 MB/sec execute 554 sec latency 24838.456 ms 2024-04-25T18:59:16.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.276+0000 7f64faa00700 1 -- 172.21.15.63:0/4189365347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4101ad0 msgr2=0x7f64f41117c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.276+0000 7f64faa00700 1 --2- 172.21.15.63:0/4189365347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4101ad0 0x7f64f41117c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00b0a0 tx=0x7f64ec00b3b0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 -- 172.21.15.63:0/4189365347 shutdown_connections 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 --2- 172.21.15.63:0/4189365347 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4103950 0x7f64f4103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 --2- 172.21.15.63:0/4189365347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4101ad0 0x7f64f41117c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 --2- 172.21.15.63:0/4189365347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f41014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 -- 172.21.15.63:0/4189365347 >> 172.21.15.63:0/4189365347 conn(0x7f64f4077fb0 msgr2=0x7f64f40ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 -- 172.21.15.63:0/4189365347 shutdown_connections 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.277+0000 7f64faa00700 1 -- 172.21.15.63:0/4189365347 wait complete. 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 Processor -- start 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 -- start start 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 0x7f64f419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 0x7f64f419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:16.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4114e50 con 0x7f64f4103950 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4114cd0 con 0x7f64f4101030 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.278+0000 7f64faa00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4114fd0 con 0x7f64f4101ad0 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f8a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 0x7f64f419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 0x7f64f419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33348/0 (socket says 172.21.15.63:33348) 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 -- 172.21.15.63:0/3700022045 learned_addr learned my addr 172.21.15.63:0/3700022045 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 -- 172.21.15.63:0/3700022045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 msgr2=0x7f64f419b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 0x7f64f419b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 -- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 msgr2=0x7f64f419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 0x7f64f419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3e00700 1 -- 172.21.15.63:0/3700022045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f41a0310 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f3400700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 0x7f64f419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.279+0000 7f64f8a00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 0x7f64f419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64f3e00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e9c0 tx=0x7f64e400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e400cbd0 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64e400ee60 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f41a0600 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e40055c0 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.280+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f41a7e80 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.281+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f4102040 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.283+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f64e4004750 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.285+0000 7f64f0c00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 0x7f64d80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.285+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8631+0+0 (secure 0 0 0) 0x7f64e40d0aa0 con 0x7f64f4101030 2024-04-25T18:59:16.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.285+0000 7f64f3400700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 0x7f64d80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:16.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.286+0000 7f64f3400700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 0x7f64d80a6270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009680 tx=0x7f64e0009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:16.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.287+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64e409d070 con 0x7f64f4101030 2024-04-25T18:59:16.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.07 MB/sec execute 555 sec latency 25838.576 ms 2024-04-25T18:59:16.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.577+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64f402df80 con 0x7f64f4101030 2024-04-25T18:59:16.579 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:16 smithi063 ceph-mon[28006]: pgmap v399: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:16.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.578+0000 7f64f0c00700 1 -- 172.21.15.63:0/3700022045 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11877 (secure 0 0 0) 0x7f64e409c970 con 0x7f64f4101030 2024-04-25T18:59:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T18:59:16.580 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:58:44.004081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 msgr2=0x7f64d80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 0x7f64d80a6270 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009680 tx=0x7f64e0009260 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 msgr2=0x7f64f419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f64e400e9c0 tx=0x7f64e400ecd0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 shutdown_connections 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f64d80a3db0 0x7f64d80a6270 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64f4103950 0x7f64f419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f64f4101030 0x7f64f419b300 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 --2- 172.21.15.63:0/3700022045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f64f4101ad0 0x7f64f419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 >> 172.21.15.63:0/3700022045 conn(0x7f64f4077fb0 msgr2=0x7f64f40fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.581+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 shutdown_connections 2024-04-25T18:59:16.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:16.582+0000 7f64faa00700 1 -- 172.21.15.63:0/3700022045 wait complete. 2024-04-25T18:59:16.584 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 48 2024-04-25T18:59:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:16 smithi114 ceph-mon[32988]: pgmap v399: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:16 smithi070 ceph-mon[32820]: pgmap v399: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.289+0000 7f49c4200640 1 -- 172.21.15.63:0/606039715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc076da0 msgr2=0x7f49bc1128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.289+0000 7f49c4200640 1 --2- 172.21.15.63:0/606039715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc076da0 0x7f49bc1128a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009900 tx=0x7f49a802a410 comp rx=0 tx=0).stop 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 -- 172.21.15.63:0/606039715 shutdown_connections 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 --2- 172.21.15.63:0/606039715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc112de0 0x7f49bc1151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 --2- 172.21.15.63:0/606039715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc076da0 0x7f49bc1128a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 --2- 172.21.15.63:0/606039715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 0x7f49bc076860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 -- 172.21.15.63:0/606039715 >> 172.21.15.63:0/606039715 conn(0x7f49bc0ffc50 msgr2=0x7f49bc102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 -- 172.21.15.63:0/606039715 shutdown_connections 2024-04-25T18:59:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.290+0000 7f49c4200640 1 -- 172.21.15.63:0/606039715 wait complete. 2024-04-25T18:59:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.291+0000 7f49c4200640 1 Processor -- start 2024-04-25T18:59:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.291+0000 7f49c4200640 1 -- start start 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 0x7f49bc1a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc112de0 0x7f49bc19eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc118830 con 0x7f49bc112de0 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1186b0 con 0x7f49bc076da0 2024-04-25T18:59:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c4200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1189b0 con 0x7f49bc076480 2024-04-25T18:59:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c0e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c2200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc112de0 0x7f49bc19eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c1800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 0x7f49bc1a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c0e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33368/0 (socket says 172.21.15.63:33368) 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.292+0000 7f49c0e00640 1 -- 172.21.15.63:0/1249683397 learned_addr learned my addr 172.21.15.63:0/1249683397 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 -- 172.21.15.63:0/1249683397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 msgr2=0x7f49bc1a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 0x7f49bc1a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 -- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc112de0 msgr2=0x7f49bc19eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc112de0 0x7f49bc19eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 -- 172.21.15.63:0/1249683397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49bc19f270 con 0x7f49bc076da0 2024-04-25T18:59:17.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.293+0000 7f49c0e00640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f49a802aa00 tx=0x7f49a8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.294+0000 7f49ba000640 1 -- 172.21.15.63:0/1249683397 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a803f070 con 0x7f49bc076da0 2024-04-25T18:59:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.294+0000 7f49ba000640 1 -- 172.21.15.63:0/1249683397 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49a803bb20 con 0x7f49bc076da0 2024-04-25T18:59:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.294+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49bc19f500 con 0x7f49bc076da0 2024-04-25T18:59:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.294+0000 7f49ba000640 1 -- 172.21.15.63:0/1249683397 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8035030 con 0x7f49bc076da0 2024-04-25T18:59:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.294+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49bc1ab800 con 0x7f49bc076da0 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.297+0000 7f49c71ab740 1 Processor -- start 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.297+0000 7f49c71ab740 1 -- start start 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.297+0000 7f49ba000640 1 -- 172.21.15.63:0/1249683397 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f49a8042070 con 0x7f49bc076da0 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 0x5617b4d34c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d41df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617b4c9a900 con 0x5617b4d35140 2024-04-25T18:59:17.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617b4c40ef0 con 0x5617b4d3fa00 2024-04-25T18:59:17.300 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c71ab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617b4c968c0 con 0x5617b4d34820 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c0e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d41df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51830/0 (socket says 172.21.15.63:51830) 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c1800640 1 -- 172.21.15.63:0/1672258648 learned_addr learned my addr 172.21.15.63:0/1672258648 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.298+0000 7f49c2200640 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 0x5617b4d34c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 -- 172.21.15.63:0/1672258648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 msgr2=0x5617b4d41df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d41df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 -- 172.21.15.63:0/1672258648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 msgr2=0x5617b4d3f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 -- 172.21.15.63:0/1672258648 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b49ad210 con 0x5617b4d34820 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c1800640 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49c2200640 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 0x5617b4d34c00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f49b4005fd0 tx=0x7f49b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deedf4b43503c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.299+0000 7f49ba000640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49aea00640 1 -- 172.21.15.63:0/1672258648 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4004d20 con 0x5617b4d34820 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49ba000640 1 -- 172.21.15.63:0/1249683397 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 8631+0+0 (secure 0 0 0) 0x7f49a80f0ea0 con 0x7f49bc076da0 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49aea00640 1 -- 172.21.15.63:0/1672258648 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b4004050 con 0x5617b4d34820 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49aea00640 1 -- 172.21.15.63:0/1672258648 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b4005220 con 0x5617b4d34820 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49c1800640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49c71ab740 1 -- 172.21.15.63:0/1672258648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 msgr2=0x5617b4d34c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 0x5617b4d34c00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f49b4005fd0 tx=0x7f49b4011810 comp rx=0 tx=0).stop 2024-04-25T18:59:17.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.300+0000 7f49c1800640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 0x7f49a40a6300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x5617b4d35dc0 tx=0x7f49b0005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 -- 172.21.15.63:0/1672258648 shutdown_connections 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d41df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d35140 0x5617b4d3f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1672258648 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d34820 0x5617b4d34c00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 -- 172.21.15.63:0/1672258648 >> 172.21.15.63:0/1672258648 conn(0x5617b4d30d30 msgr2=0x5617b4d31110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 -- 172.21.15.63:0/1672258648 shutdown_connections 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 -- 172.21.15.63:0/1672258648 wait complete. 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 -- start start 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.301+0000 7f49c71ab740 1 Processor -- start 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d3ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d3d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617b4d42630 con 0x5617b4d34820 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5617b4c54310 con 0x5617b4d3fa00 2024-04-25T18:59:17.307 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5617b4d427b0 con 0x5617b4d35140 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c71ab740 10 client.0 _ll_register_callbacks cb 0x5617b4c22cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38816/0 (socket says 172.21.15.63:38816) 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c1800640 1 -- 172.21.15.63:0/4159667563 learned_addr learned my addr 172.21.15.63:0/4159667563 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c0e00640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d3ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.302+0000 7f49c2200640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d3d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 -- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 msgr2=0x5617b4d3d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d3d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 -- 172.21.15.63:0/4159667563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 msgr2=0x5617b4d3c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 -- 172.21.15.63:0/4159667563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b4a79620 con 0x5617b4d35140 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c1800640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:17.308 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c2200640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d3d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c0e00640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d3ceb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f49a80ee520 tx=0x7f49a8009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ef8e6cab2308de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49a3400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8049030 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49b4004050 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8631+0+0 (secure 0 0 0) 0x7f49a80ee7a0 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f49a80bc960 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c71ab740 10 client.26728 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.303+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5617b4d42680 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b4004050 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f49a80bd080 con 0x5617b4d35140 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 10 client.26728 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 10 client.26728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 20 client.26728 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:17.309 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4d4fc50 0x5617b4d520d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 4 client.26728 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617b4cae8b0 con 0x5617b4d4fc50 2024-04-25T18:59:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.304+0000 7f49c1800640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4d4fc50 0x5617b4d520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.305+0000 7f49c1800640 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4d4fc50 0x5617b4d520d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=708e4376d5a72119 server_cookie=c8133b27acd7ada2 in_seq=0 out_seq=0 2024-04-25T18:59:17.310 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.305+0000 7f49a3400640 10 client.26728 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49a3400640 1 -- 172.21.15.63:0/4159667563 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617b4cae8b0 con 0x5617b4d4fc50 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49a3400640 10 client.26728 handle_command_reply: tid=0 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 client.26728 shutdown 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 msgr2=0x5617b4d3ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d3ceb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f49a80ee520 tx=0x7f49a8009560 comp rx=0 tx=0).stop 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 shutdown_connections 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4d4fc50 0x5617b4d520d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d3d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d3ceb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49a3400640 0 client.26728 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 --2- 172.21.15.63:0/4159667563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d3c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.306+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 >> 172.21.15.63:0/4159667563 conn(0x5617b4d30d30 msgr2=0x5617b4d36ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.307+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 shutdown_connections 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.307+0000 7f49c71ab740 1 -- 172.21.15.63:0/4159667563 wait complete. 2024-04-25T18:59:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.307+0000 7f49c71ab740 20 client.26728 trim_cache size 0 max 16384 2024-04-25T18:59:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.324+0000 7f49c71ab740 1 Processor -- start 2024-04-25T18:59:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.324+0000 7f49c71ab740 1 -- start start 2024-04-25T18:59:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.324+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d57ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c71ab740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b6c80 con 0x5617b4d34820 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c71ab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b4490 con 0x5617b4d3fa00 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c71ab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b2300 con 0x5617b4d35140 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38818/0 (socket says 172.21.15.63:38818) 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c2200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d57ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 -- 172.21.15.63:0/1784051779 learned_addr learned my addr 172.21.15.63:0/1784051779 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55228/0 (socket says 172.21.15.63:55228) 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 -- 172.21.15.63:0/1784051779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 msgr2=0x5617b4d579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 -- 172.21.15.63:0/1784051779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 msgr2=0x5617b4d57ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d57ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c1800640 1 -- 172.21.15.63:0/1784051779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b4d3af00 con 0x5617b4d34820 2024-04-25T18:59:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.325+0000 7f49c0e00640 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.326+0000 7f49c1800640 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009ab0 tx=0x7f49b0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669eb4ae6056f5cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.326+0000 7f49ae000640 1 -- 172.21.15.63:0/1784051779 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b002b130 con 0x5617b4d34820 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.326+0000 7f49ae000640 1 -- 172.21.15.63:0/1784051779 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b0031070 con 0x5617b4d34820 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.326+0000 7f49ae000640 1 -- 172.21.15.63:0/1784051779 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49b001dc30 con 0x5617b4d34820 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 -- 172.21.15.63:0/1784051779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 msgr2=0x5617b4d4b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009ab0 tx=0x7f49b0019000 comp rx=0 tx=0).stop 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 -- 172.21.15.63:0/1784051779 shutdown_connections 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d57ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 --2- 172.21.15.63:0/1784051779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d4b4f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 -- 172.21.15.63:0/1784051779 >> 172.21.15.63:0/1784051779 conn(0x5617b4d30d30 msgr2=0x5617b4d31110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 -- 172.21.15.63:0/1784051779 shutdown_connections 2024-04-25T18:59:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.327+0000 7f49c71ab740 1 -- 172.21.15.63:0/1784051779 wait complete. 2024-04-25T18:59:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 -- start start 2024-04-25T18:59:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 Processor -- start 2024-04-25T18:59:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d47f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d45110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b2950 con 0x5617b4d34820 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b3dc0 con 0x5617b4d3fa00 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f49bc1b3020 con 0x5617b4d35140 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.328+0000 7f49c71ab740 10 client.0 _ll_register_callbacks cb 0x7f49a8009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d47f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c1800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d47f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38834/0 (socket says 172.21.15.63:38834) 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c1800640 1 -- 172.21.15.63:0/3662410318 learned_addr learned my addr 172.21.15.63:0/3662410318 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c2200640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d45110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 -- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 msgr2=0x5617b4d45110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d45110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 -- 172.21.15.63:0/3662410318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 msgr2=0x5617b4d47f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d47f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 -- 172.21.15.63:0/3662410318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b4db4210 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49c0e00640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d449d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f49a8031650 tx=0x7f49a80c59e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c32df455ef53b9e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.329+0000 7f49a3400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8049070 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 8631+0+0 (secure 0 0 0) 0x7f49a80c5750 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49b0031070 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f49a80db050 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 10 client.26734 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5617b4db52d0 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b0031070 con 0x5617b4d35140 2024-04-25T18:59:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f49a8031b70 con 0x5617b4d35140 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 10 client.26734 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 10 client.26734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.330+0000 7f49c71ab740 20 client.26734 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4db5940 0x5617b4db7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49c71ab740 4 client.26734 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617b4ca8320 con 0x5617b4db5940 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49c1800640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4db5940 0x5617b4db7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49c1800640 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4db5940 0x5617b4db7d60 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=585e3cf54470a0b5 server_cookie=580c14c3a21fad92 in_seq=0 out_seq=0 2024-04-25T18:59:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.331+0000 7f49a3400640 10 client.26734 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.353+0000 7f49a3400640 1 -- 172.21.15.63:0/3662410318 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5617b4ca8320 con 0x5617b4db5940 2024-04-25T18:59:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.353+0000 7f49a3400640 10 client.26734 handle_command_reply: tid=0 2024-04-25T18:59:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.353+0000 7f49c71ab740 1 client.26734 shutdown 2024-04-25T18:59:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.353+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 msgr2=0x5617b4d449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.353+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d449d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f49a8031650 tx=0x7f49a80c59e0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 shutdown_connections 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5617b4db5940 0x5617b4db7d60 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5617b4d3fa00 0x5617b4d45110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5617b4d35140 0x5617b4d449d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49a3400640 0 client.26734 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 --2- 172.21.15.63:0/3662410318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617b4d34820 0x5617b4d47f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 >> 172.21.15.63:0/3662410318 conn(0x5617b4d30d30 msgr2=0x5617b4d3b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 shutdown_connections 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 1 -- 172.21.15.63:0/3662410318 wait complete. 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.354+0000 7f49c71ab740 20 client.26734 trim_cache size 0 max 16384 2024-04-25T18:59:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:59:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:59:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:59:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:59:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 msgr2=0x7f49a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 0x7f49a40a6300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x5617b4d35dc0 tx=0x7f49b0005f90 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 msgr2=0x7f49bc1a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f49a802aa00 tx=0x7f49a8003c10 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 shutdown_connections 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f49a40a3e40 0x7f49a40a6300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49bc076da0 0x7f49bc1a4e80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f49bc112de0 0x7f49bc19eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 --2- 172.21.15.63:0/1249683397 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f49bc076480 0x7f49bc1a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 >> 172.21.15.63:0/1249683397 conn(0x7f49bc0ffc50 msgr2=0x7f49bc1143c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 shutdown_connections 2024-04-25T18:59:17.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:17.355+0000 7f49c4200640 1 -- 172.21.15.63:0/1249683397 wait complete. 2024-04-25T18:59:17.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.07 MB/sec execute 556 sec latency 26838.688 ms 2024-04-25T18:59:17.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:17 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3700022045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:59:17.614 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id 446cb4ea --release --await' 2024-04-25T18:59:17.614 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id 446cb4ea --release --await 2024-04-25T18:59:17.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:17 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3700022045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:59:17.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:17 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3700022045' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T18:59:18.077 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T18:59:18.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.06 MB/sec execute 557 sec latency 27838.817 ms 2024-04-25T18:59:18.760 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:18 smithi063 ceph-mon[28006]: pgmap v400: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:18 smithi114 ceph-mon[32988]: pgmap v400: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:18.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:18 smithi070 ceph-mon[32820]: pgmap v400: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 24 op/s 2024-04-25T18:59:19.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.06 MB/sec execute 558 sec latency 28838.954 ms 2024-04-25T18:59:20.282 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:20 smithi063 ceph-mon[28006]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T18:59:20.282 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:20 smithi063 ceph-mon[28006]: pgmap v402: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:59:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:20 smithi114 ceph-mon[32988]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T18:59:20.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:20 smithi114 ceph-mon[32988]: pgmap v402: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:59:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:20 smithi070 ceph-mon[32820]: osdmap e113: 12 total, 12 up, 12 in 2024-04-25T18:59:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:20 smithi070 ceph-mon[32820]: pgmap v402: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 38 KiB/s wr, 14 op/s 2024-04-25T18:59:20.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 381258 3.05 MB/sec execute 559 sec latency 29839.086 ms 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 -- 172.21.15.63:0/1324847444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c102af0 msgr2=0x7f032c102f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 --2- 172.21.15.63:0/1324847444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c102af0 0x7f032c102f70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0318009920 tx=0x7f0318009c30 comp rx=0 tx=0).stop 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 -- 172.21.15.63:0/1324847444 shutdown_connections 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 --2- 172.21.15.63:0/1324847444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c10f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 --2- 172.21.15.63:0/1324847444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c107940 0x7f032c107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 --2- 172.21.15.63:0/1324847444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c102af0 0x7f032c102f70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 -- 172.21.15.63:0/1324847444 >> 172.21.15.63:0/1324847444 conn(0x7f032c079ba0 msgr2=0x7f032c079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 -- 172.21.15.63:0/1324847444 shutdown_connections 2024-04-25T18:59:20.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.576+0000 7f0330a00700 1 -- 172.21.15.63:0/1324847444 wait complete. 2024-04-25T18:59:20.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.578+0000 7f0330a00700 1 Processor -- start 2024-04-25T18:59:20.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.578+0000 7f0330a00700 1 -- start start 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 0x7f032c19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 0x7f032c19fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f032c112c90 con 0x7f032c102af0 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f032c112b10 con 0x7f032c107940 2024-04-25T18:59:20.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0330a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f032c112e10 con 0x7f032c1034b0 2024-04-25T18:59:20.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0329e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 0x7f032c19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0329400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f032a800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 0x7f032c19fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0329400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55250/0 (socket says 172.21.15.63:55250) 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0329400700 1 -- 172.21.15.63:0/1011953778 learned_addr learned my addr 172.21.15.63:0/1011953778 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.579+0000 7f0329e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 0x7f032c19b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38862/0 (socket says 172.21.15.63:38862) 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 -- 172.21.15.63:0/1011953778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 msgr2=0x7f032c19fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 0x7f032c19fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 -- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 msgr2=0x7f032c19b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 0x7f032c19b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 -- 172.21.15.63:0/1011953778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c1a0410 con 0x7f032c1034b0 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f032a800700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 0x7f032c19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:20.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0329400700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0318000c00 tx=0x7f031800f990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0322a00700 1 -- 172.21.15.63:0/1011953778 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318027070 con 0x7f032c1034b0 2024-04-25T18:59:20.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0322a00700 1 -- 172.21.15.63:0/1011953778 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03180055f0 con 0x7f032c1034b0 2024-04-25T18:59:20.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.580+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c1a06a0 con 0x7f032c1034b0 2024-04-25T18:59:20.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.581+0000 7f0322a00700 1 -- 172.21.15.63:0/1011953778 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318004930 con 0x7f032c1034b0 2024-04-25T18:59:20.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.581+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c1a7f80 con 0x7f032c1034b0 2024-04-25T18:59:20.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.584+0000 7f0322a00700 1 -- 172.21.15.63:0/1011953778 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f031802a070 con 0x7f032c1034b0 2024-04-25T18:59:20.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.586+0000 7f033b3a4b80 1 Processor -- start 2024-04-25T18:59:20.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.586+0000 7f033b3a4b80 1 -- start start 2024-04-25T18:59:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a5f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f0322a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 0x7f03100a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a69f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a6c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5582579b1f90 con 0x558257a6a4d0 2024-04-25T18:59:20.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558257958830 con 0x558257a5f660 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5582579adf50 con 0x558257a5ed40 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f0322a00700 1 -- 172.21.15.63:0/1011953778 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f03180d10f0 con 0x7f032c1034b0 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f032a800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a5f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f032a800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a5f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:55256/0 (socket says 172.21.15.63:55256) 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f032a800700 1 -- 172.21.15.63:0/2932012252 learned_addr learned my addr 172.21.15.63:0/2932012252 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f032a800700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 0x7f03100a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f0329e00700 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a69f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.587+0000 7f0329400700 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a6c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 -- 172.21.15.63:0/2932012252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 msgr2=0x558257a5f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a5f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 -- 172.21.15.63:0/2932012252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 msgr2=0x558257a6c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a6c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 -- 172.21.15.63:0/2932012252 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825794e780 con 0x558257a5f660 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0329e00700 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a69f90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f031c009cc0 tx=0x7f031c0058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0fa62e4c8ff8e84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f032a800700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 0x7f03100a62f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0324009550 tx=0x7f032400b040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0317400700 1 -- 172.21.15.63:0/2932012252 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c011ce0 con 0x558257a5f660 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0317400700 1 -- 172.21.15.63:0/2932012252 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f031c004500 con 0x558257a5f660 2024-04-25T18:59:20.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.588+0000 7f0317400700 1 -- 172.21.15.63:0/2932012252 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c005280 con 0x558257a5f660 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2932012252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 msgr2=0x558257a69f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a69f90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f031c009cc0 tx=0x7f031c0058e0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2932012252 shutdown_connections 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a5f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a6c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2932012252 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a69f90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2932012252 >> 172.21.15.63:0/2932012252 conn(0x55825798e050 msgr2=0x558257a5b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2932012252 shutdown_connections 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.589+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2932012252 wait complete. 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.590+0000 7f033b3a4b80 1 -- start start 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.590+0000 7f033b3a4b80 1 Processor -- start 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257a66e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558257a6cf60 con 0x558257a6a4d0 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55825796b960 con 0x558257a5ed40 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558257a6d0e0 con 0x558257a5f660 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f033b3a4b80 10 client.0 _ll_register_callbacks cb 0x55825794a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46176/0 (socket says 172.21.15.63:46176) 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 -- 172.21.15.63:0/3924101743 learned_addr learned my addr 172.21.15.63:0/3924101743 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f032a800700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329400700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257a66e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 -- 172.21.15.63:0/3924101743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 msgr2=0x558257a66e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257a66e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 -- 172.21.15.63:0/3924101743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 msgr2=0x558257a673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.591+0000 7f0329e00700 1 -- 172.21.15.63:0/3924101743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582578ff9d0 con 0x558257a5ed40 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f032a800700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f0329e00700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f031c009830 tx=0x7f031c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=681ba5743da9e290 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f030be00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c026020 con 0x558257a5ed40 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f031c004500 con 0x558257a5ed40 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f031c004750 con 0x558257a5ed40 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.592+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f031c031050 con 0x558257a5ed40 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 10 client.26726 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:20.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558257948bf0 con 0x558257a5ed40 2024-04-25T18:59:20.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f031c004500 con 0x558257a5ed40 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f031c018070 con 0x558257a5ed40 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 10 client.26726 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 10 client.26726 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 20 client.26726 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a79a60 0x558257a7be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 4 client.26726 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.593+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582579bf780 con 0x558257a79a60 2024-04-25T18:59:20.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.594+0000 7f0329400700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a79a60 0x558257a7be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.594+0000 7f0329400700 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a79a60 0x558257a7be50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d576217729ad5e server_cookie=dc440b8f7d993868 in_seq=0 out_seq=0 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.594+0000 7f030be00700 10 client.26726 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f030be00700 1 -- 172.21.15.63:0/3924101743 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582579bf780 con 0x558257a79a60 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f030be00700 10 client.26726 handle_command_reply: tid=0 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 client.26726 shutdown 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 msgr2=0x558257a69b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f031c009830 tx=0x7f031c005e80 comp rx=0 tx=0).stop 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 shutdown_connections 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a79a60 0x558257a7be50 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a6a4d0 0x558257a673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5ed40 0x558257a69b20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3924101743 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257a66e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 >> 172.21.15.63:0/3924101743 conn(0x55825798e050 msgr2=0x558257a5ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.595+0000 7f030be00700 0 client.26726 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.596+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 shutdown_connections 2024-04-25T18:59:20.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.596+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3924101743 wait complete. 2024-04-25T18:59:20.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.596+0000 7f033b3a4b80 20 client.26726 trim_cache size 0 max 16384 2024-04-25T18:59:20.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.617+0000 7f033b3a4b80 1 Processor -- start 2024-04-25T18:59:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.617+0000 7f033b3a4b80 1 -- start start 2024-04-25T18:59:20.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a68d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a5b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1b1f20 con 0x55825798e050 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1af6f0 con 0x558257a5f660 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1ad560 con 0x558257a5ed40 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f0329e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f0329400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a68d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f0329e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38890/0 (socket says 172.21.15.63:38890) 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f032a800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a5b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.618+0000 7f0329e00700 1 -- 172.21.15.63:0/3342277450 learned_addr learned my addr 172.21.15.63:0/3342277450 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 -- 172.21.15.63:0/3342277450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 msgr2=0x558257a5b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a5b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 -- 172.21.15.63:0/3342277450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 msgr2=0x558257a65e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 -- 172.21.15.63:0/3342277450 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558257a5fce0 con 0x558257a5ed40 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329e00700 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0329400700 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a68d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f03180ce7a0 tx=0x7f0318019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69c6b68d4319673 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0316a00700 1 -- 172.21.15.63:0/3342277450 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031809d080 con 0x558257a5ed40 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0316a00700 1 -- 172.21.15.63:0/3342277450 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03180a1430 con 0x558257a5ed40 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.619+0000 7f0316a00700 1 -- 172.21.15.63:0/3342277450 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03180cea20 con 0x558257a5ed40 2024-04-25T18:59:20.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3342277450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 msgr2=0x558257a68d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a68d30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f03180ce7a0 tx=0x7f0318019040 comp rx=0 tx=0).stop 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3342277450 shutdown_connections 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558257a5f660 0x558257a5b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825798e050 0x558257a65e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/3342277450 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5ed40 0x558257a68d30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3342277450 >> 172.21.15.63:0/3342277450 conn(0x558257a6cae0 msgr2=0x558257a5ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3342277450 shutdown_connections 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- 172.21.15.63:0/3342277450 wait complete. 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.620+0000 7f033b3a4b80 1 -- start start 2024-04-25T18:59:20.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 Processor -- start 2024-04-25T18:59:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 0x558257a6fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257adfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1adbb0 con 0x558257a5ed40 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1af020 con 0x55825798e050 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1ae280 con 0x558257a5f660 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.621+0000 7f033b3a4b80 10 client.0 _ll_register_callbacks cb 0x55825794a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 0x558257a6fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:46196/0 (socket says 172.21.15.63:46196) 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 -- 172.21.15.63:0/2540917265 learned_addr learned my addr 172.21.15.63:0/2540917265 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 0x558257a6fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38898/0 (socket says 172.21.15.63:38898) 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f032a800700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257adfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 -- 172.21.15.63:0/2540917265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 msgr2=0x558257adfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257adfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 -- 172.21.15.63:0/2540917265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 msgr2=0x558257a6fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:20.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 0x558257a6fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 -- 172.21.15.63:0/2540917265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558257ae03d0 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f0329e00700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f031c000c00 tx=0x7f031c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14167c90d03e8b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.622+0000 7f030be00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c026040 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03180a1430 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f031c0053a0 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f031c004100 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 10 client.26732 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558257ae16d0 con 0x55825798e050 2024-04-25T18:59:20.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03180a1430 con 0x55825798e050 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f031c03c030 con 0x55825798e050 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 10 client.26732 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 10 client.26732 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 20 client.26732 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a6a4d0 0x558257ae3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 4 client.26732 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "446cb4ea", "release": true, "await": true}] 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.623+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "446cb4ea", "release": true, "await": true}) -- 0x5582579b9820 con 0x558257a6a4d0 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.624+0000 7f0329400700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a6a4d0 0x558257ae3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.624+0000 7f0329400700 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a6a4d0 0x558257ae3de0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f3c59684933a995 server_cookie=8b0c3245ef42b468 in_seq=0 out_seq=0 2024-04-25T18:59:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.624+0000 7f030be00700 10 client.26732 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:20.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.630+0000 7f030be00700 1 -- 172.21.15.63:0/2540917265 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x5582579b9820 con 0x558257a6a4d0 2024-04-25T18:59:20.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.630+0000 7f030be00700 10 client.26732 handle_command_reply: tid=0 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.630+0000 7f033b3a4b80 1 client.26732 shutdown 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 msgr2=0x558257a6f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f031c000c00 tx=0x7f031c009580 comp rx=0 tx=0).stop 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 shutdown_connections 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558257a6a4d0 0x558257ae3de0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558257a5f660 0x558257adfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825798e050 0x558257a6f520 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 --2- 172.21.15.63:0/2540917265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558257a5ed40 0x558257a6fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 >> 172.21.15.63:0/2540917265 conn(0x558257a6cae0 msgr2=0x558257a74e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f030be00700 0 client.26732 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 shutdown_connections 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 1 -- 172.21.15.63:0/2540917265 wait complete. 2024-04-25T18:59:20.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.631+0000 7f033b3a4b80 20 client.26732 trim_cache size 0 max 16384 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 48, 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 40, 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "446cb4ea": { 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 40, 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:59:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 81.5, 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T18:59:20.635 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 msgr2=0x7f03100a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 0x7f03100a62f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0324009550 tx=0x7f032400b040 comp rx=0 tx=0).stop 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 msgr2=0x7f032c19b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0318000c00 tx=0x7f031800f990 comp rx=0 tx=0).stop 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 shutdown_connections 2024-04-25T18:59:20.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f03100a3e30 0x7f03100a62f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f032c107940 0x7f032c19fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f032c102af0 0x7f032c19b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 --2- 172.21.15.63:0/1011953778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f032c1034b0 0x7f032c19b940 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 >> 172.21.15.63:0/1011953778 conn(0x7f032c079ba0 msgr2=0x7f032c100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 shutdown_connections 2024-04-25T18:59:20.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T18:59:20.632+0000 7f0330a00700 1 -- 172.21.15.63:0/1011953778 wait complete. 2024-04-25T18:59:21.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 382492 3.06 MB/sec execute 560 sec latency 29955.826 ms 2024-04-25T18:59:21.743 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '446cb4ea', seconds elapsed: 38.4 2024-04-25T18:59:22.522 INFO:tasks.workunit.client.0.smithi063.stdout: 1 383480 3.06 MB/sec execute 561 sec latency 15.828 ms 2024-04-25T18:59:22.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:22 smithi114 ceph-mon[32988]: pgmap v403: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:22.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:22 smithi063 ceph-mon[28006]: pgmap v403: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:22 smithi070 ceph-mon[32820]: pgmap v403: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:23.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 384198 3.06 MB/sec execute 562 sec latency 22.121 ms 2024-04-25T18:59:24.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 384796 3.06 MB/sec execute 563 sec latency 422.771 ms 2024-04-25T18:59:24.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:24 smithi114 ceph-mon[32988]: pgmap v404: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:24 smithi063 ceph-mon[28006]: pgmap v404: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:24 smithi070 ceph-mon[32820]: pgmap v404: 97 pgs: 97 active+clean; 109 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 37 KiB/s wr, 14 op/s 2024-04-25T18:59:25.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 384796 3.06 MB/sec execute 564 sec latency 1422.901 ms 2024-04-25T18:59:26.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 385577 3.06 MB/sec execute 565 sec latency 1792.000 ms 2024-04-25T18:59:26.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:26 smithi114 ceph-mon[32988]: pgmap v405: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:26.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:26 smithi063 ceph-mon[28006]: pgmap v405: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:26 smithi070 ceph-mon[32820]: pgmap v405: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:27.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 386897 3.07 MB/sec execute 566 sec latency 16.467 ms 2024-04-25T18:59:28.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 387400 3.07 MB/sec execute 567 sec latency 21.819 ms 2024-04-25T18:59:28.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:28 smithi063 ceph-mon[28006]: pgmap v406: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:28.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:28 smithi070 ceph-mon[32820]: pgmap v406: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:28 smithi114 ceph-mon[32988]: pgmap v406: 97 pgs: 97 active+clean; 112 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 422 KiB/s wr, 62 op/s 2024-04-25T18:59:29.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 388335 3.07 MB/sec execute 568 sec latency 29.159 ms 2024-04-25T18:59:30.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:30 smithi063 ceph-mon[28006]: pgmap v407: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 102 op/s 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 -- 172.21.15.63:0/4282667880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14103830 msgr2=0x7f3c1410bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 --2- 172.21.15.63:0/4282667880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14103830 0x7f3c1410bd10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009860 tx=0x7f3c0002a3d0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 -- 172.21.15.63:0/4282667880 shutdown_connections 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 --2- 172.21.15.63:0/4282667880 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c1410c330 0x7f3c1410e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 --2- 172.21.15.63:0/4282667880 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c14103830 0x7f3c1410bd10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 --2- 172.21.15.63:0/4282667880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14102f10 0x7f3c141032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 -- 172.21.15.63:0/4282667880 >> 172.21.15.63:0/4282667880 conn(0x7f3c140fc770 msgr2=0x7f3c140feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.502+0000 7f3c19600640 1 -- 172.21.15.63:0/4282667880 shutdown_connections 2024-04-25T18:59:30.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.503+0000 7f3c19600640 1 -- 172.21.15.63:0/4282667880 wait complete. 2024-04-25T18:59:30.507 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.503+0000 7f3c19600640 1 Processor -- start 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 -- start start 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 0x7f3c141a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 0x7f3c141a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 0x7f3c1419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14111ff0 con 0x7f3c14103830 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14111e70 con 0x7f3c14102f10 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c19600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14112170 con 0x7f3c1410c330 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c13400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 0x7f3c141a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c13e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 0x7f3c141a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 0x7f3c1419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c13400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 0x7f3c141a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44498/0 (socket says 172.21.15.63:44498) 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.504+0000 7f3c13400640 1 -- 172.21.15.63:0/2120940895 learned_addr learned my addr 172.21.15.63:0/2120940895 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:30.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 -- 172.21.15.63:0/2120940895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 msgr2=0x7f3c1419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 0x7f3c1419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 -- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 msgr2=0x7f3c141a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 0x7f3c141a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 -- 172.21.15.63:0/2120940895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1419aef0 con 0x7f3c14102f10 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c18c00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 0x7f3c1419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 0x7f3c141a0570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3c04005fd0 tx=0x7f3c0400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c10c00640 1 -- 172.21.15.63:0/2120940895 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400cd00 con 0x7f3c14102f10 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1419b1e0 con 0x7f3c14102f10 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c10c00640 1 -- 172.21.15.63:0/2120940895 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c040035f0 con 0x7f3c14102f10 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.505+0000 7f3c10c00640 1 -- 172.21.15.63:0/2120940895 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04010670 con 0x7f3c14102f10 2024-04-25T18:59:30.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.506+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c141a74f0 con 0x7f3c14102f10 2024-04-25T18:59:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.510+0000 7f3c1d9c7740 1 Processor -- start 2024-04-25T18:59:30.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.510+0000 7f3c1d9c7740 1 -- start start 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337061650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x563337063f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c10c00640 1 -- 172.21.15.63:0/2120940895 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f3c04003aa0 con 0x7f3c14102f10 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563336fbc900 con 0x563337061b90 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563336f62ef0 con 0x5633370569b0 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563336fb88c0 con 0x5633370572d0 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60670/0 (socket says 172.21.15.63:60670) 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c18c00640 1 -- 172.21.15.63:0/2511259779 learned_addr learned my addr 172.21.15.63:0/2511259779 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c13400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x563337063f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.511+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337061650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 -- 172.21.15.63:0/2511259779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 msgr2=0x563337061650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337061650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 -- 172.21.15.63:0/2511259779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 msgr2=0x563337063f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x563337063f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 -- 172.21.15.63:0/2511259779 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563336ccf210 con 0x5633370569b0 2024-04-25T18:59:30.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337061650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:30.519 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c18c00640 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c011790 tx=0x7f3c0c011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b3bc6f1d62e851 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c09200640 1 -- 172.21.15.63:0/2511259779 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c003c30 con 0x5633370569b0 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c09200640 1 -- 172.21.15.63:0/2511259779 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c00fe50 con 0x5633370569b0 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c09200640 1 -- 172.21.15.63:0/2511259779 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c0193e0 con 0x5633370569b0 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.512+0000 7f3c10c00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 0x7f3bf80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c10c00640 1 -- 172.21.15.63:0/2120940895 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f3c040d09d0 con 0x7f3c14102f10 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 0x7f3bf80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/2511259779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 msgr2=0x563337056d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c011790 tx=0x7f3c0c011c50 comp rx=0 tx=0).stop 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/2511259779 shutdown_connections 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x563337063f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337061650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/2511259779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337056d90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.520 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/2511259779 >> 172.21.15.63:0/2511259779 conn(0x563337052ec0 msgr2=0x5633370532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/2511259779 shutdown_connections 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/2511259779 wait complete. 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.513+0000 7f3c13e00640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 0x7f3bf80a6250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3c040ce190 tx=0x7f3c040d4030 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.514+0000 7f3c1d9c7740 1 -- start start 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.514+0000 7f3c1d9c7740 1 Processor -- start 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333705eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333705f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5633370647c0 con 0x5633370569b0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563336f76310 con 0x563337061b90 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563337064940 con 0x5633370572d0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c1d9c7740 10 client.0 _ll_register_callbacks cb 0x563336f44cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333705eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.521 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39140/0 (socket says 172.21.15.63:39140) 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 -- 172.21.15.63:0/1869748990 learned_addr learned my addr 172.21.15.63:0/1869748990 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c18c00640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333705f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 -- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 msgr2=0x56333705f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333705f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 -- 172.21.15.63:0/1869748990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 msgr2=0x56333705eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333705eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c13400640 1 -- 172.21.15.63:0/1869748990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563336d9b620 con 0x5633370572d0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.515+0000 7f3c18c00640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333705eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3c13400640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c00030bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bed69f1afdaf88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00005240 con 0x5633370572d0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c0c00fe50 con 0x5633370572d0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f3c000053e0 con 0x5633370572d0 2024-04-25T18:59:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3c000055e0 con 0x5633370572d0 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3c1d9c7740 10 client.26764 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.516+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x563337070a20 con 0x5633370572d0 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c00fe50 con 0x5633370572d0 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f3c00046660 con 0x5633370572d0 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 10 client.26764 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 10 client.26764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 20 client.26764 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563337071e90 0x5633370742f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 4 client.26764 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563336fd08b0 con 0x563337071e90 2024-04-25T18:59:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.517+0000 7f3c18c00640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563337071e90 0x5633370742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.518+0000 7f3c18c00640 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563337071e90 0x5633370742f0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1f9444c002f814 server_cookie=babb2f33c3460dc3 in_seq=0 out_seq=0 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.518+0000 7f3bfd600640 10 client.26764 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3bfd600640 1 -- 172.21.15.63:0/1869748990 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563336fd08b0 con 0x563337071e90 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3bfd600640 10 client.26764 handle_command_reply: tid=0 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 client.26764 shutdown 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 msgr2=0x56333705f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3c00000c00 tx=0x7f3c00030bb0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 shutdown_connections 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563337071e90 0x5633370742f0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333705f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x56333705f020 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/1869748990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333705eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3bfd600640 0 client.26764 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:30.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 >> 172.21.15.63:0/1869748990 conn(0x563337052ec0 msgr2=0x563337058bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 shutdown_connections 2024-04-25T18:59:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.519+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/1869748990 wait complete. 2024-04-25T18:59:30.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.520+0000 7f3c1d9c7740 20 client.26764 trim_cache size 0 max 16384 2024-04-25T18:59:30.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 388368 3.06 MB/sec execute 569 sec latency 973.096 ms 2024-04-25T18:59:30.538 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 Processor -- start 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 -- start start 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337079c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333707a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141b2750 con 0x5633370569b0 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141aff60 con 0x563337061b90 2024-04-25T18:59:30.539 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.536+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141addd0 con 0x5633370572d0 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44516/0 (socket says 172.21.15.63:44516) 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 -- 172.21.15.63:0/3604998256 learned_addr learned my addr 172.21.15.63:0/3604998256 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c13400640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337079c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333707a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 -- 172.21.15.63:0/3604998256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 msgr2=0x563337079c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337079c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 -- 172.21.15.63:0/3604998256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 msgr2=0x56333707a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333707a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c18c00640 1 -- 172.21.15.63:0/3604998256 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56333705a680 con 0x5633370569b0 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.537+0000 7f3c13400640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337079c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c18c00640 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x56333707a680 tx=0x7f3c0c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe4f0c19763fe11 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3bffe00640 1 -- 172.21.15.63:0/3604998256 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c01f040 con 0x5633370569b0 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3bffe00640 1 -- 172.21.15.63:0/3604998256 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c011050 con 0x5633370569b0 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3bffe00640 1 -- 172.21.15.63:0/3604998256 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c004950 con 0x5633370569b0 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/3604998256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 msgr2=0x56333706d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x56333707a680 tx=0x7f3c0c009500 comp rx=0 tx=0).stop 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/3604998256 shutdown_connections 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563337061b90 0x56333707a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x563337079c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/3604998256 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633370569b0 0x56333706d680 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.538+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/3604998256 >> 172.21.15.63:0/3604998256 conn(0x563337052ec0 msgr2=0x5633370532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.539+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/3604998256 shutdown_connections 2024-04-25T18:59:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.539+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/3604998256 wait complete. 2024-04-25T18:59:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.539+0000 7f3c1d9c7740 1 -- start start 2024-04-25T18:59:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.539+0000 7f3c1d9c7740 1 Processor -- start 2024-04-25T18:59:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337067ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x5633370681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x5633370d7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141ae420 con 0x563337061b90 2024-04-25T18:59:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141af890 con 0x5633370569b0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3c141aeaf0 con 0x5633370572d0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c1d9c7740 10 client.0 _ll_register_callbacks cb 0x7f3c000307d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337067ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c18c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337067ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60700/0 (socket says 172.21.15.63:60700) 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c18c00640 1 -- 172.21.15.63:0/437616251 learned_addr learned my addr 172.21.15.63:0/437616251 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x5633370681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c18c00640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x5633370d7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 -- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 msgr2=0x563337067ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337067ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 -- 172.21.15.63:0/437616251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 msgr2=0x5633370d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x5633370d7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.540+0000 7f3c13400640 1 -- 172.21.15.63:0/437616251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633370d79c0 con 0x5633370572d0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3c13400640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x5633370681e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009830 tx=0x7f3c0002a3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=640107956f2f1ee4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00052070 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c0c011050 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f3c000037f0 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f3c00009450 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3c1d9c7740 10 client.26770 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.541+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5633370d8c70 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c0c011050 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f3c0003c540 con 0x5633370572d0 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 10 client.26770 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 10 client.26770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 20 client.26770 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5633370d9420 0x5633370db840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 4 client.26770 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563336fca320 con 0x5633370d9420 2024-04-25T18:59:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.542+0000 7f3c18c00640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5633370d9420 0x5633370db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.543+0000 7f3c18c00640 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5633370d9420 0x5633370db840 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca5746c326b0227 server_cookie=1a84fda7b83ea76b in_seq=0 out_seq=0 2024-04-25T18:59:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.543+0000 7f3bfd600640 10 client.26770 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.571+0000 7f3bfd600640 1 -- 172.21.15.63:0/437616251 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563336fca320 con 0x5633370d9420 2024-04-25T18:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.571+0000 7f3bfd600640 10 client.26770 handle_command_reply: tid=0 2024-04-25T18:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.571+0000 7f3c1d9c7740 1 client.26770 shutdown 2024-04-25T18:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.571+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 msgr2=0x5633370681e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.571+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x5633370681e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009830 tx=0x7f3c0002a3d0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 shutdown_connections 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5633370d9420 0x5633370db840 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563337061b90 0x5633370d7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5633370572d0 0x5633370681e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 --2- 172.21.15.63:0/437616251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5633370569b0 0x563337067ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3bfd600640 0 client.26770 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 >> 172.21.15.63:0/437616251 conn(0x563337052ec0 msgr2=0x563337058360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 shutdown_connections 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 1 -- 172.21.15.63:0/437616251 wait complete. 2024-04-25T18:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.572+0000 7f3c1d9c7740 20 client.26770 trim_cache size 0 max 16384 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 msgr2=0x7f3bf80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 0x7f3bf80a6250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3c040ce190 tx=0x7f3c040d4030 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 msgr2=0x7f3c141a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 0x7f3c141a0570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3c04005fd0 tx=0x7f3c0400ee50 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 shutdown_connections 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f3bf80a3d90 0x7f3bf80a6250 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3c1410c330 0x7f3c1419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c14103830 0x7f3c141a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 --2- 172.21.15.63:0/2120940895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3c14102f10 0x7f3c141a0570 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 >> 172.21.15.63:0/2120940895 conn(0x7f3c140fc770 msgr2=0x7f3c14101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 shutdown_connections 2024-04-25T18:59:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:30.573+0000 7f3c19600640 1 -- 172.21.15.63:0/2120940895 wait complete. 2024-04-25T18:59:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:30 smithi070 ceph-mon[32820]: pgmap v407: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 102 op/s 2024-04-25T18:59:30.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:30 smithi114 ceph-mon[32988]: pgmap v407: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 760 KiB/s wr, 102 op/s 2024-04-25T18:59:31.523 INFO:tasks.workunit.client.0.smithi063.stdout: 1 389590 3.07 MB/sec execute 570 sec latency 1019.903 ms 2024-04-25T18:59:32.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 390546 3.08 MB/sec execute 571 sec latency 19.498 ms 2024-04-25T18:59:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:32 smithi063 ceph-mon[28006]: pgmap v408: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:32 smithi070 ceph-mon[32820]: pgmap v408: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:32 smithi114 ceph-mon[32988]: pgmap v408: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:33.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 391146 3.07 MB/sec execute 572 sec latency 24.604 ms 2024-04-25T18:59:34.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 391911 3.08 MB/sec execute 573 sec latency 172.774 ms 2024-04-25T18:59:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:34 smithi063 ceph-mon[28006]: pgmap v409: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:34 smithi070 ceph-mon[32820]: pgmap v409: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:34 smithi114 ceph-mon[32988]: pgmap v409: 97 pgs: 97 active+clean; 116 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 645 KiB/s wr, 84 op/s 2024-04-25T18:59:35.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 391911 3.07 MB/sec execute 574 sec latency 1172.934 ms 2024-04-25T18:59:36.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 392710 3.07 MB/sec execute 575 sec latency 1551.601 ms 2024-04-25T18:59:36.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:36 smithi063 ceph-mon[28006]: pgmap v410: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-04-25T18:59:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:36 smithi070 ceph-mon[32820]: pgmap v410: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-04-25T18:59:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:36 smithi114 ceph-mon[32988]: pgmap v410: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 MiB/s wr, 146 op/s 2024-04-25T18:59:37.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 393974 3.08 MB/sec execute 576 sec latency 16.084 ms 2024-04-25T18:59:38.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 394361 3.08 MB/sec execute 577 sec latency 25.157 ms 2024-04-25T18:59:38.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:38 smithi063 ceph-mon[28006]: pgmap v411: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 700 KiB/s wr, 102 op/s 2024-04-25T18:59:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:38 smithi070 ceph-mon[32820]: pgmap v411: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 700 KiB/s wr, 102 op/s 2024-04-25T18:59:38.816 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:38 smithi114 ceph-mon[32988]: pgmap v411: 97 pgs: 97 active+clean; 115 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 700 KiB/s wr, 102 op/s 2024-04-25T18:59:39.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 395220 3.08 MB/sec execute 578 sec latency 27.833 ms 2024-04-25T18:59:40.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:40 smithi114 ceph-mon[32988]: pgmap v412: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-04-25T18:59:40.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:40 smithi063 ceph-mon[28006]: pgmap v412: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-04-25T18:59:40.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:40 smithi070 ceph-mon[32820]: pgmap v412: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-04-25T18:59:40.524 INFO:tasks.workunit.client.0.smithi063.stdout: 1 395459 3.08 MB/sec execute 579 sec latency 606.992 ms 2024-04-25T18:59:41.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 396246 3.08 MB/sec execute 580 sec latency 985.346 ms 2024-04-25T18:59:42.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 397255 3.09 MB/sec execute 581 sec latency 232.342 ms 2024-04-25T18:59:42.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:42 smithi114 ceph-mon[32988]: pgmap v413: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:42.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:42 smithi063 ceph-mon[28006]: pgmap v413: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:42 smithi070 ceph-mon[32820]: pgmap v413: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:43.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 397853 3.09 MB/sec execute 582 sec latency 15.583 ms 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 -- 172.21.15.63:0/254356580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f501043c0 msgr2=0x7f7f501047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 --2- 172.21.15.63:0/254356580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f501043c0 0x7f7f501047a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009860 tx=0x7f7f4002a340 comp rx=0 tx=0).stop 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 -- 172.21.15.63:0/254356580 shutdown_connections 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 --2- 172.21.15.63:0/254356580 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f50109490 0x7f7f50111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 --2- 172.21.15.63:0/254356580 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f50104d70 0x7f7f50108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 --2- 172.21.15.63:0/254356580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f501043c0 0x7f7f501047a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.727+0000 7f7f55600640 1 -- 172.21.15.63:0/254356580 >> 172.21.15.63:0/254356580 conn(0x7f7f500ffc20 msgr2=0x7f7f50102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.728+0000 7f7f55600640 1 -- 172.21.15.63:0/254356580 shutdown_connections 2024-04-25T18:59:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.728+0000 7f7f55600640 1 -- 172.21.15.63:0/254356580 wait complete. 2024-04-25T18:59:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.729+0000 7f7f55600640 1 Processor -- start 2024-04-25T18:59:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.729+0000 7f7f55600640 1 -- start start 2024-04-25T18:59:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f50109490 0x7f7f501a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f5019a740 0x7f7f5019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50114420 con 0x7f7f50109490 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501142a0 con 0x7f7f501043c0 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f55600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501145a0 con 0x7f7f5019a740 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f47e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f47e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56204/0 (socket says 172.21.15.63:56204) 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f47e00640 1 -- 172.21.15.63:0/3828703179 learned_addr learned my addr 172.21.15.63:0/3828703179 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f4f400640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f5019a740 0x7f7f5019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.730+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f50109490 0x7f7f501a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 -- 172.21.15.63:0/3828703179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f5019a740 msgr2=0x7f7f5019abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f5019a740 0x7f7f5019abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 -- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f50109490 msgr2=0x7f7f501a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f50109490 0x7f7f501a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 -- 172.21.15.63:0/3828703179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5019b340 con 0x7f7f501043c0 2024-04-25T18:59:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47e00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7f4002a8b0 tx=0x7f7f40005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47400640 1 -- 172.21.15.63:0/3828703179 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40037040 con 0x7f7f501043c0 2024-04-25T18:59:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47400640 1 -- 172.21.15.63:0/3828703179 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f40041070 con 0x7f7f501043c0 2024-04-25T18:59:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f47400640 1 -- 172.21.15.63:0/3828703179 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40004cd0 con 0x7f7f501043c0 2024-04-25T18:59:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.731+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5019b5d0 con 0x7f7f501043c0 2024-04-25T18:59:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.732+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f501a7700 con 0x7f7f501043c0 2024-04-25T18:59:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f47400640 1 -- 172.21.15.63:0/3828703179 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f7f40004380 con 0x7f7f501043c0 2024-04-25T18:59:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f58592740 1 Processor -- start 2024-04-25T18:59:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f58592740 1 -- start start 2024-04-25T18:59:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f58592740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f58592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.735+0000 7f7f58592740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 0x55c1b55cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f58592740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1b5526900 con 0x55c1b55cba00 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f58592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b54ccef0 con 0x55c1b55c1140 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f58592740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1b55228c0 con 0x55c1b55c0820 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54892/0 (socket says 172.21.15.63:54892) 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 -- 172.21.15.63:0/2488864813 learned_addr learned my addr 172.21.15.63:0/2488864813 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f47e00640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 0x55c1b55cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 -- 172.21.15.63:0/2488864813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 msgr2=0x55c1b55cb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55cb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 -- 172.21.15.63:0/2488864813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 msgr2=0x55c1b55cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 0x55c1b55cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4f400640 1 -- 172.21.15.63:0/2488864813 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b5239210 con 0x55c1b55c0820 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.736+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 0x55c1b55cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009f50 tx=0x7f7f480118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512c48da2015f6e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f47400640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 0x7f7f340a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f3f400640 1 -- 172.21.15.63:0/2488864813 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48004d20 con 0x55c1b55c0820 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f3f400640 1 -- 172.21.15.63:0/2488864813 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f48004050 con 0x55c1b55c0820 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f3f400640 1 -- 172.21.15.63:0/2488864813 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48005240 con 0x55c1b55c0820 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f47400640 1 -- 172.21.15.63:0/3828703179 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f7f400f01a0 con 0x7f7f501043c0 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.737+0000 7f7f47e00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 0x7f7f340a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- 172.21.15.63:0/2488864813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 msgr2=0x55c1b55c0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009f50 tx=0x7f7f480118d0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f47e00640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 0x7f7f340a6350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55c1b55c1dc0 tx=0x7f7f400f3c50 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- 172.21.15.63:0/2488864813 shutdown_connections 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55cb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55cba00 0x55c1b55cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 --2- 172.21.15.63:0/2488864813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c0c00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- 172.21.15.63:0/2488864813 >> 172.21.15.63:0/2488864813 conn(0x55c1b55bcd30 msgr2=0x55c1b55bd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- 172.21.15.63:0/2488864813 shutdown_connections 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- 172.21.15.63:0/2488864813 wait complete. 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.738+0000 7f7f58592740 1 -- start start 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 Processor -- start 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1b55ce630 con 0x55c1b55c1140 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1b54e0310 con 0x55c1b55cba00 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1b55ce7b0 con 0x55c1b55c0820 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f58592740 10 client.0 _ll_register_callbacks cb 0x55c1b54aecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54902/0 (socket says 172.21.15.63:54902) 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f4f400640 1 -- 172.21.15.63:0/2943540604 learned_addr learned my addr 172.21.15.63:0/2943540604 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.739+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 -- 172.21.15.63:0/2943540604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 msgr2=0x55c1b55c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 -- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 msgr2=0x55c1b55c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 -- 172.21.15.63:0/2943540604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b5305620 con 0x55c1b55c1140 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55c8e90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55c1b55be150 tx=0x7f7f380035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cfa97c9fc51a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T18:59:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.740+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3801b670 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f7f3801b810 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f48004050 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7f3801ba10 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 10 client.16572 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55c1b55da860 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f48004050 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f7f38011450 con 0x55c1b55c1140 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 10 client.16572 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 20 client.16572 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b55dbdc0 0x55c1b55de1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 4 client.16572 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.741+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1b553a8b0 con 0x55c1b55dbdc0 2024-04-25T18:59:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.742+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b55dbdc0 0x55c1b55de1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.742+0000 7f7f4f400640 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b55dbdc0 0x55c1b55de1f0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7ef6e8e10cb587f server_cookie=eca585fe1b6d7a4c in_seq=0 out_seq=0 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.742+0000 7f7f33e00640 10 client.16572 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f33e00640 1 -- 172.21.15.63:0/2943540604 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1b553a8b0 con 0x55c1b55dbdc0 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f33e00640 10 client.16572 handle_command_reply: tid=0 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 client.16572 shutdown 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 msgr2=0x55c1b55c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55c8e90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55c1b55be150 tx=0x7f7f380035f0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 shutdown_connections 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b55dbdc0 0x55c1b55de1f0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55c8e90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 --2- 172.21.15.63:0/2943540604 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 >> 172.21.15.63:0/2943540604 conn(0x55c1b55bcd30 msgr2=0x55c1b55c2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f33e00640 0 client.16572 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 shutdown_connections 2024-04-25T18:59:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.743+0000 7f7f58592740 1 -- 172.21.15.63:0/2943540604 wait complete. 2024-04-25T18:59:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.744+0000 7f7f58592740 20 client.16572 trim_cache size 0 max 16384 2024-04-25T18:59:43.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.762+0000 7f7f58592740 1 Processor -- start 2024-04-25T18:59:43.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.762+0000 7f7f58592740 1 -- start start 2024-04-25T18:59:43.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c0820 0x55c1b55d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55cba00 0x55c1b55e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501b27d0 con 0x55c1b55c0820 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501affe0 con 0x55c1b55c1140 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f58592740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501ade50 con 0x55c1b55cba00 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f4ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c0820 0x55c1b55d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f4ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56216/0 (socket says 172.21.15.63:56216) 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f4ea00640 1 -- 172.21.15.63:0/1744013188 learned_addr learned my addr 172.21.15.63:0/1744013188 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.763+0000 7f7f4f400640 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55cba00 0x55c1b55e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 -- 172.21.15.63:0/1744013188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55cba00 msgr2=0x55c1b55e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55cba00 0x55c1b55e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 -- 172.21.15.63:0/1744013188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c0820 msgr2=0x55c1b55d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c0820 0x55c1b55d74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 -- 172.21.15.63:0/1744013188 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b55c18d0 con 0x55c1b55c1140 2024-04-25T18:59:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7f38001eb0 tx=0x7f7f38005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9564fe3ddd50f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f3ea00640 1 -- 172.21.15.63:0/1744013188 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38031070 con 0x55c1b55c1140 2024-04-25T18:59:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f3ea00640 1 -- 172.21.15.63:0/1744013188 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f38016070 con 0x55c1b55c1140 2024-04-25T18:59:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.764+0000 7f7f3ea00640 1 -- 172.21.15.63:0/1744013188 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38005030 con 0x55c1b55c1140 2024-04-25T18:59:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 -- 172.21.15.63:0/1744013188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 msgr2=0x55c1b55bc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7f38001eb0 tx=0x7f7f38005a40 comp rx=0 tx=0).stop 2024-04-25T18:59:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 -- 172.21.15.63:0/1744013188 shutdown_connections 2024-04-25T18:59:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55c1140 0x55c1b55bc610 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55cba00 0x55c1b55e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 --2- 172.21.15.63:0/1744013188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c0820 0x55c1b55d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 -- 172.21.15.63:0/1744013188 >> 172.21.15.63:0/1744013188 conn(0x55c1b55bcd30 msgr2=0x55c1b55c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 -- 172.21.15.63:0/1744013188 shutdown_connections 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.765+0000 7f7f58592740 1 -- 172.21.15.63:0/1744013188 wait complete. 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.766+0000 7f7f58592740 1 -- start start 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.766+0000 7f7f58592740 1 Processor -- start 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55d3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501ae4a0 con 0x55c1b55c1140 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501af910 con 0x55c1b55cba00 2024-04-25T18:59:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7f501aeb70 con 0x55c1b55c0820 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f58592740 10 client.0 _ll_register_callbacks cb 0x7f7f38009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54918/0 (socket says 172.21.15.63:54918) 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 -- 172.21.15.63:0/3117672645 learned_addr learned my addr 172.21.15.63:0/3117672645 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55d3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 -- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 msgr2=0x55c1b55d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 -- 172.21.15.63:0/3117672645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 msgr2=0x55c1b55d3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55d3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.767+0000 7f7f4f400640 1 -- 172.21.15.63:0/3117672645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b55d13d0 con 0x55c1b55c0820 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f4f400640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f48012fd0 tx=0x7f7f48000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=485cd0d769b3f59e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f33e00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48031040 con 0x55c1b55c0820 2024-04-25T18:59:43.772 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f38016070 con 0x55c1b55c0820 2024-04-25T18:59:43.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f7f48004920 con 0x55c1b55c0820 2024-04-25T18:59:43.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7f48004b20 con 0x55c1b55c0820 2024-04-25T18:59:43.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f58592740 10 client.26788 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:43.773 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.768+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55c1b5641e30 con 0x55c1b55c0820 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f38016070 con 0x55c1b55c0820 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f7f48037040 con 0x55c1b55c0820 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 10 client.26788 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 10 client.26788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 20 client.26788 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b56423d0 0x55c1b56447f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 4 client.26788 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1b5534320 con 0x55c1b56423d0 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.769+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b56423d0 0x55c1b56447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:43.774 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.770+0000 7f7f4ea00640 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b56423d0 0x55c1b56447f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eadd60f82ca0b2bc server_cookie=9da92baba44210fc in_seq=0 out_seq=0 2024-04-25T18:59:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.770+0000 7f7f33e00640 10 client.26788 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f33e00640 1 -- 172.21.15.63:0/3117672645 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1b5534320 con 0x55c1b56423d0 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f33e00640 10 client.26788 handle_command_reply: tid=0 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 client.26788 shutdown 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 msgr2=0x55c1b55d37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f48012fd0 tx=0x7f7f48000f40 comp rx=0 tx=0).stop 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 shutdown_connections 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1b56423d0 0x55c1b56447f0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1b55cba00 0x55c1b55d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1b55c1140 0x55c1b55d3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 --2- 172.21.15.63:0/3117672645 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1b55c0820 0x55c1b55d37b0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f33e00640 0 client.26788 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.797+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 >> 172.21.15.63:0/3117672645 conn(0x55c1b55bcd30 msgr2=0x55c1b55c21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.798+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 shutdown_connections 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.798+0000 7f7f58592740 1 -- 172.21.15.63:0/3117672645 wait complete. 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.798+0000 7f7f58592740 20 client.26788 trim_cache size 0 max 16384 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:59:43.801 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 msgr2=0x7f7f340a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 0x7f7f340a6350 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x55c1b55c1dc0 tx=0x7f7f400f3c50 comp rx=0 tx=0).stop 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 msgr2=0x7f7f501a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7f4002a8b0 tx=0x7f7f40005770 comp rx=0 tx=0).stop 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 shutdown_connections 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f7f340a3e90 0x7f7f340a6350 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.802 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7f5019a740 0x7f7f5019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f50109490 0x7f7f501a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 --2- 172.21.15.63:0/3828703179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f501043c0 0x7f7f501a0680 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 >> 172.21.15.63:0/3828703179 conn(0x7f7f500ffc20 msgr2=0x7f7f50101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 shutdown_connections 2024-04-25T18:59:43.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:43.799+0000 7f7f55600640 1 -- 172.21.15.63:0/3828703179 wait complete. 2024-04-25T18:59:44.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 398799 3.09 MB/sec execute 583 sec latency 42.607 ms 2024-04-25T18:59:44.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:44 smithi114 ceph-mon[32988]: pgmap v414: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:44.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:44 smithi063 ceph-mon[28006]: pgmap v414: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:44 smithi070 ceph-mon[32820]: pgmap v414: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 768 KiB/s wr, 106 op/s 2024-04-25T18:59:45.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 399814 3.09 MB/sec execute 584 sec latency 108.978 ms 2024-04-25T18:59:46.525 INFO:tasks.workunit.client.0.smithi063.stdout: 1 400989 3.10 MB/sec execute 585 sec latency 129.193 ms 2024-04-25T18:59:46.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:46 smithi063 ceph-mon[28006]: pgmap v415: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T18:59:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:46 smithi070 ceph-mon[32820]: pgmap v415: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T18:59:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:46 smithi114 ceph-mon[32988]: pgmap v415: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T18:59:47.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 401537 3.10 MB/sec execute 586 sec latency 21.552 ms 2024-04-25T18:59:48.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 402497 3.10 MB/sec execute 587 sec latency 40.144 ms 2024-04-25T18:59:48.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:48 smithi063 ceph-mon[28006]: pgmap v416: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 725 KiB/s wr, 114 op/s 2024-04-25T18:59:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:48 smithi070 ceph-mon[32820]: pgmap v416: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 725 KiB/s wr, 114 op/s 2024-04-25T18:59:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:48 smithi114 ceph-mon[32988]: pgmap v416: 97 pgs: 97 active+clean; 119 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 725 KiB/s wr, 114 op/s 2024-04-25T18:59:49.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 403736 3.11 MB/sec execute 588 sec latency 20.927 ms 2024-04-25T18:59:50.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 404700 3.11 MB/sec execute 589 sec latency 49.337 ms 2024-04-25T18:59:50.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:50 smithi063 ceph-mon[28006]: pgmap v417: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-04-25T18:59:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:50 smithi070 ceph-mon[32820]: pgmap v417: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-04-25T18:59:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:50 smithi114 ceph-mon[32988]: pgmap v417: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-04-25T18:59:51.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 405314 3.11 MB/sec execute 590 sec latency 33.047 ms 2024-04-25T18:59:52.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 406604 3.11 MB/sec execute 591 sec latency 22.372 ms 2024-04-25T18:59:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:52 smithi063 ceph-mon[28006]: pgmap v418: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:52.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:52 smithi070 ceph-mon[32820]: pgmap v418: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:52 smithi114 ceph-mon[32988]: pgmap v418: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:53.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 407750 3.12 MB/sec execute 592 sec latency 17.428 ms 2024-04-25T18:59:54.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 408425 3.12 MB/sec execute 593 sec latency 14.952 ms 2024-04-25T18:59:54.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:54 smithi063 ceph-mon[28006]: pgmap v419: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:54 smithi070 ceph-mon[32820]: pgmap v419: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:54 smithi114 ceph-mon[32988]: pgmap v419: 97 pgs: 97 active+clean; 123 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 767 KiB/s wr, 133 op/s 2024-04-25T18:59:55.526 INFO:tasks.workunit.client.0.smithi063.stdout: 1 409352 3.13 MB/sec execute 594 sec latency 21.205 ms 2024-04-25T18:59:56.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 410466 3.13 MB/sec execute 595 sec latency 23.909 ms 2024-04-25T18:59:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:56 smithi063 ceph-mon[28006]: pgmap v420: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 202 op/s 2024-04-25T18:59:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:56 smithi070 ceph-mon[32820]: pgmap v420: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 202 op/s 2024-04-25T18:59:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:56 smithi114 ceph-mon[32988]: pgmap v420: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 202 op/s 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.955+0000 7f832ce00640 1 -- 172.21.15.63:0/3056726124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281092a0 msgr2=0x7f8328110e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.955+0000 7f832ce00640 1 --2- 172.21.15.63:0/3056726124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281092a0 0x7f8328110e70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8320009ca0 tx=0x7f832002a8c0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 -- 172.21.15.63:0/3056726124 shutdown_connections 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 --2- 172.21.15.63:0/3056726124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281092a0 0x7f8328110e70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 --2- 172.21.15.63:0/3056726124 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8328104b80 0x7f8328108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 --2- 172.21.15.63:0/3056726124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83281041d0 0x7f83281045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 -- 172.21.15.63:0/3056726124 >> 172.21.15.63:0/3056726124 conn(0x7f83280ffa70 msgr2=0x7f8328101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 -- 172.21.15.63:0/3056726124 shutdown_connections 2024-04-25T18:59:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.956+0000 7f832ce00640 1 -- 172.21.15.63:0/3056726124 wait complete. 2024-04-25T18:59:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.957+0000 7f832ce00640 1 Processor -- start 2024-04-25T18:59:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.957+0000 7f832ce00640 1 -- start start 2024-04-25T18:59:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281041d0 0x7f832819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 0x7f832819f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8328114240 con 0x7f83281041d0 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83281140c0 con 0x7f832819f020 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f832ce00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83281143c0 con 0x7f83281092a0 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f8326a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f8327e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 0x7f832819f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f8326a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:36942/0 (socket says 172.21.15.63:36942) 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f8326a00640 1 -- 172.21.15.63:0/3644945329 learned_addr learned my addr 172.21.15.63:0/3644945329 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.958+0000 7f8327400640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281041d0 0x7f832819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 -- 172.21.15.63:0/3644945329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 msgr2=0x7f832819aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 -- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281041d0 msgr2=0x7f832819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281041d0 0x7f832819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 -- 172.21.15.63:0/3644945329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832819fb10 con 0x7f832819f020 2024-04-25T18:59:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8326a00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f8327e00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 0x7f832819f4d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8320005e30 tx=0x7f83200319f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.959+0000 7f831fe00640 1 -- 172.21.15.63:0/3644945329 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320003bf0 con 0x7f832819f020 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.960+0000 7f831fe00640 1 -- 172.21.15.63:0/3644945329 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8320003d90 con 0x7f832819f020 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.960+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832819fda0 con 0x7f832819f020 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.960+0000 7f831fe00640 1 -- 172.21.15.63:0/3644945329 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832002fba0 con 0x7f832819f020 2024-04-25T18:59:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.960+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a75f0 con 0x7f832819f020 2024-04-25T18:59:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.964+0000 7f8331225740 1 Processor -- start 2024-04-25T18:59:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.964+0000 7f831fe00640 1 -- 172.21.15.63:0/3644945329 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f832002fd40 con 0x7f832819f020 2024-04-25T18:59:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 -- start start 2024-04-25T18:59:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec82d0 0x5584faed2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5584fae2d900 con 0x5584faec82d0 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584fadd3ef0 con 0x5584faec79b0 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8331225740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584fae298c0 con 0x5584faed2b90 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44324/0 (socket says 172.21.15.63:44324) 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8326a00640 1 -- 172.21.15.63:0/1058780320 learned_addr learned my addr 172.21.15.63:0/1058780320 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8326a00640 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.965+0000 7f8327400640 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec82d0 0x5584faed2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 -- 172.21.15.63:0/1058780320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 msgr2=0x5584faed4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 -- 172.21.15.63:0/1058780320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec82d0 msgr2=0x5584faed2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec82d0 0x5584faed2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 -- 172.21.15.63:0/1058780320 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584fab40210 con 0x5584faec79b0 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8326a00640 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5584faec5050 tx=0x7f8310005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0596b041b9fb36b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f831fe00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 0x7f830c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f831fe00640 1 -- 172.21.15.63:0/3644945329 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f83200f1a80 con 0x7f832819f020 2024-04-25T18:59:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8327400640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 0x7f830c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f831cc00640 1 -- 172.21.15.63:0/1058780320 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83100120d0 con 0x5584faec79b0 2024-04-25T18:59:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f831cc00640 1 -- 172.21.15.63:0/1058780320 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310012c50 con 0x5584faec79b0 2024-04-25T18:59:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f831cc00640 1 -- 172.21.15.63:0/1058780320 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310010c00 con 0x5584faec79b0 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8331225740 1 -- 172.21.15.63:0/1058780320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 msgr2=0x5584faec7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8331225740 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x5584faec5050 tx=0x7f8310005d10 comp rx=0 tx=0).stop 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.967+0000 7f8327400640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 0x7f830c0a6250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x5584faec8f50 tx=0x7f831800b390 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 -- 172.21.15.63:0/1058780320 shutdown_connections 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec82d0 0x5584faed2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 --2- 172.21.15.63:0/1058780320 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec79b0 0x5584faec7d90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 -- 172.21.15.63:0/1058780320 >> 172.21.15.63:0/1058780320 conn(0x5584faec3ec0 msgr2=0x5584faec42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 -- 172.21.15.63:0/1058780320 shutdown_connections 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 -- 172.21.15.63:0/1058780320 wait complete. 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 -- start start 2024-04-25T18:59:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.968+0000 7f8331225740 1 Processor -- start 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faecfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faec82d0 0x5584faed0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5584faed5760 con 0x5584faec79b0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5584fade7310 con 0x5584faed2b90 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5584faed58e0 con 0x5584faec82d0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8331225740 10 client.0 _ll_register_callbacks cb 0x5584fadb5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8327e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8327e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44334/0 (socket says 172.21.15.63:44334) 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8327e00640 1 -- 172.21.15.63:0/3254944919 learned_addr learned my addr 172.21.15.63:0/3254944919 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8327400640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faecfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.969+0000 7f8326a00640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faec82d0 0x5584faed0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 -- 172.21.15.63:0/3254944919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faec82d0 msgr2=0x5584faed0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faec82d0 0x5584faed0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 -- 172.21.15.63:0/3254944919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 msgr2=0x5584faecfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faecfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 -- 172.21.15.63:0/3254944919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584fac0c620 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327400640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faecfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8327e00640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f83200b9fe0 tx=0x7f83200bd4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ab1132d94077fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8315600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83200c1f30 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f83200ba1f0 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8310012c50 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f83200c6760 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8331225740 10 client.26780 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.970+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5584faed5760 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310012c50 con 0x5584faed2b90 2024-04-25T18:59:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f83200c62d0 con 0x5584faed2b90 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 10 client.26780 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 10 client.26780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 20 client.26780 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faee3010 0x5584faee5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 4 client.26780 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584fae418b0 con 0x5584faee3010 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.971+0000 7f8327400640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faee3010 0x5584faee5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.972+0000 7f8327400640 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faee3010 0x5584faee5450 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31ca4082e3aeb59 server_cookie=4f558f48b70edc6e in_seq=0 out_seq=0 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.972+0000 7f8315600640 10 client.26780 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:56.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8315600640 1 -- 172.21.15.63:0/3254944919 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584fae418b0 con 0x5584faee3010 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8315600640 10 client.26780 handle_command_reply: tid=0 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 client.26780 shutdown 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 msgr2=0x5584faed05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f83200b9fe0 tx=0x7f83200bd4e0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 shutdown_connections 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faee3010 0x5584faee5450 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faed2b90 0x5584faed05d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faec82d0 0x5584faed0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 --2- 172.21.15.63:0/3254944919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faecfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 >> 172.21.15.63:0/3254944919 conn(0x5584faec3ec0 msgr2=0x5584faec9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8315600640 0 client.26780 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 shutdown_connections 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 1 -- 172.21.15.63:0/3254944919 wait complete. 2024-04-25T18:59:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.973+0000 7f8331225740 20 client.26780 trim_cache size 0 max 16384 2024-04-25T18:59:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 Processor -- start 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 -- start start 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faede6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83281b25d0 con 0x5584faec79b0 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83281afde0 con 0x5584faec82d0 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.991+0000 7f8331225740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83281adc50 con 0x5584faed2b90 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:36968/0 (socket says 172.21.15.63:36968) 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44336/0 (socket says 172.21.15.63:44336) 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327e00640 1 -- 172.21.15.63:0/4206585624 learned_addr learned my addr 172.21.15.63:0/4206585624 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327400640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faede6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 -- 172.21.15.63:0/4206585624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 msgr2=0x5584faed0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 -- 172.21.15.63:0/4206585624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 msgr2=0x5584faede6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faede6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8326a00640 1 -- 172.21.15.63:0/4206585624 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584faeca900 con 0x5584faec82d0 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327400640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faede6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.992+0000 7f8327e00640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.993+0000 7f8326a00640 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5584faecf3b0 tx=0x7f8310005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=685a497b5db4e1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.993+0000 7f8317e00640 1 -- 172.21.15.63:0/4206585624 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831000ebb0 con 0x5584faec82d0 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.993+0000 7f8317e00640 1 -- 172.21.15.63:0/4206585624 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310023070 con 0x5584faec82d0 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.993+0000 7f8317e00640 1 -- 172.21.15.63:0/4206585624 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83100038e0 con 0x5584faec82d0 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- 172.21.15.63:0/4206585624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 msgr2=0x5584faeca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x5584faecf3b0 tx=0x7f8310005850 comp rx=0 tx=0).stop 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- 172.21.15.63:0/4206585624 shutdown_connections 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faeca3c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faed0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 --2- 172.21.15.63:0/4206585624 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faede6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- 172.21.15.63:0/4206585624 >> 172.21.15.63:0/4206585624 conn(0x5584faec3ec0 msgr2=0x5584faecfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- 172.21.15.63:0/4206585624 shutdown_connections 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- 172.21.15.63:0/4206585624 wait complete. 2024-04-25T18:59:56.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.994+0000 7f8331225740 1 -- start start 2024-04-25T18:59:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 Processor -- start 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faed7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faf415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83281ae2a0 con 0x5584faec79b0 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83281af710 con 0x5584faec82d0 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83281ae970 con 0x5584faed2b90 2024-04-25T18:59:57.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8331225740 10 client.0 _ll_register_callbacks cb 0x7f83200bd950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8327400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faed7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8327e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faf415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.995+0000 7f8326a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44350/0 (socket says 172.21.15.63:44350) 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 -- 172.21.15.63:0/3165987139 learned_addr learned my addr 172.21.15.63:0/3165987139 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T18:59:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 -- 172.21.15.63:0/3165987139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 msgr2=0x5584faf415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faf415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 -- 172.21.15.63:0/3165987139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 msgr2=0x5584faed7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faed7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 -- 172.21.15.63:0/3165987139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584faf41c10 con 0x5584faec82d0 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8327e00640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faf415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:57.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8327400640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faed7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8326a00640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8310012b60 tx=0x7f831000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e89fb040f945d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8315600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.996+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310030060 con 0x5584faec82d0 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 8587+0+0 (secure 0 0 0) 0x7f83100052e0 con 0x5584faec82d0 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8310023070 con 0x5584faec82d0 2024-04-25T18:59:57.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f831000bba0 con 0x5584faec82d0 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 10 client.26786 fetch_fsmap learned FSMap version 48 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5584faf48fd0 con 0x5584faec82d0 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8310023070 con 0x5584faec82d0 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f83100104f0 con 0x5584faec82d0 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 10 client.26786 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 10 client.26786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 20 client.26786 populate_metadata read hostname 'smithi063' 2024-04-25T18:59:57.008 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faf494d0 0x5584faf4b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T18:59:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 4 client.26786 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T18:59:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.997+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584fae3b320 con 0x5584faf494d0 2024-04-25T18:59:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.998+0000 7f8327400640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faf494d0 0x5584faf4b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T18:59:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.998+0000 7f8327400640 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faf494d0 0x5584faf4b8f0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=652139d53c34668c server_cookie=160007125fa0db11 in_seq=0 out_seq=0 2024-04-25T18:59:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:56.998+0000 7f8315600640 10 client.26786 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8315600640 1 -- 172.21.15.63:0/3165987139 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584fae3b320 con 0x5584faf494d0 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8315600640 10 client.26786 handle_command_reply: tid=0 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 client.26786 shutdown 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 msgr2=0x5584faed83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8310012b60 tx=0x7f831000ee60 comp rx=0 tx=0).stop 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 shutdown_connections 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5584faf494d0 0x5584faf4b8f0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5584faec82d0 0x5584faed83b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5584faed2b90 0x5584faf415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 --2- 172.21.15.63:0/3165987139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5584faec79b0 0x5584faed7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.037 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8315600640 0 client.26786 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 >> 172.21.15.63:0/3165987139 conn(0x5584faec3ec0 msgr2=0x5584faecfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 shutdown_connections 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.034+0000 7f8331225740 1 -- 172.21.15.63:0/3165987139 wait complete. 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.035+0000 7f8331225740 20 client.26786 trim_cache size 0 max 16384 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T18:59:57.038 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T18:59:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T18:59:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 msgr2=0x7f830c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 0x7f830c0a6250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x5584faec8f50 tx=0x7f831800b390 comp rx=0 tx=0).stop 2024-04-25T18:59:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 msgr2=0x7f832819f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T18:59:57.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 0x7f832819f4d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8320005e30 tx=0x7f83200319f0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 shutdown_connections 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f830c0a3d90 0x7f830c0a6250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f832819f020 0x7f832819f4d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83281092a0 0x7f832819aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 --2- 172.21.15.63:0/3644945329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83281041d0 0x7f832819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 >> 172.21.15.63:0/3644945329 conn(0x7f83280ffa70 msgr2=0x7f8328101d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 shutdown_connections 2024-04-25T18:59:57.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T18:59:57.036+0000 7f832ce00640 1 -- 172.21.15.63:0/3644945329 wait complete. 2024-04-25T18:59:57.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 411707 3.14 MB/sec execute 596 sec latency 33.741 ms 2024-04-25T18:59:58.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 412214 3.14 MB/sec execute 597 sec latency 23.730 ms 2024-04-25T18:59:58.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 18:59:58 smithi063 ceph-mon[28006]: pgmap v421: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-25T18:59:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 18:59:58 smithi070 ceph-mon[32820]: pgmap v421: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-25T18:59:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 18:59:58 smithi114 ceph-mon[32988]: pgmap v421: 97 pgs: 97 active+clean; 127 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-25T18:59:59.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 413183 3.14 MB/sec execute 598 sec latency 20.148 ms 2024-04-25T19:00:00.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 414211 3.14 MB/sec execute 599 sec latency 99.413 ms 2024-04-25T19:00:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:00 smithi063 ceph-mon[28006]: pgmap v422: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-04-25T19:00:00.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:00 smithi063 ceph-mon[28006]: overall HEALTH_OK 2024-04-25T19:00:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:00 smithi070 ceph-mon[32820]: pgmap v422: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-04-25T19:00:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:00 smithi070 ceph-mon[32820]: overall HEALTH_OK 2024-04-25T19:00:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:00 smithi114 ceph-mon[32988]: pgmap v422: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 222 op/s 2024-04-25T19:00:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:00 smithi114 ceph-mon[32988]: overall HEALTH_OK 2024-04-25T19:00:01.527 INFO:tasks.workunit.client.0.smithi063.stdout: 1 cleanup 600 sec 2024-04-25T19:00:01.960 INFO:tasks.workunit.client.0.smithi063.stdout: 0 cleanup 601 sec 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Operation Count AvgLat MaxLat 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: ---------------------------------------- 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: NTCreateX 60464 5.049 46542.376 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Close 44431 0.012 0.550 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Rename 2566 15.424 4212.504 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Unlink 12191 6.609 29794.597 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Qpathinfo 54895 1.730 13402.908 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Qfileinfo 9585 0.005 0.370 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Qfsinfo 10038 0.480 25.687 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Sfileinfo 4932 5.325 25537.395 2024-04-25T19:00:01.961 INFO:tasks.workunit.client.0.smithi063.stdout: Find 21181 0.441 34.148 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: WriteX 29895 0.194 101.168 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: ReadX 95043 0.072 232.309 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: LockX 198 0.953 3.205 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: UnlockX 198 0.788 3.314 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: Flush 4229 5.336 55.392 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stdout:Throughput 3.14357 MB/sec 1 clients 1 procs max_latency=46542.837 ms 2024-04-25T19:00:01.962 INFO:tasks.workunit.client.0.smithi063.stderr:+ dbench 10 2024-04-25T19:00:01.964 INFO:tasks.workunit.client.0.smithi063.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-25T19:00:01.964 INFO:tasks.workunit.client.0.smithi063.stdout: 2024-04-25T19:00:01.964 INFO:tasks.workunit.client.0.smithi063.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-25T19:00:01.968 INFO:tasks.workunit.client.0.smithi063.stdout:9 of 10 processes prepared for launch 0 sec 2024-04-25T19:00:02.069 INFO:tasks.workunit.client.0.smithi063.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-25T19:00:02.069 INFO:tasks.workunit.client.0.smithi063.stdout:releasing clients 2024-04-25T19:00:02.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:02 smithi063 ceph-mon[28006]: pgmap v423: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:02.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:02 smithi070 ceph-mon[32820]: pgmap v423: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:02 smithi114 ceph-mon[32988]: pgmap v423: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:03.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 339 122.38 MB/sec warmup 1 sec latency 623.015 ms 2024-04-25T19:00:04.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 750 99.31 MB/sec warmup 2 sec latency 1623.166 ms 2024-04-25T19:00:04.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:04 smithi070 ceph-mon[32820]: pgmap v424: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:04 smithi114 ceph-mon[32988]: pgmap v424: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:04 smithi063 ceph-mon[28006]: pgmap v424: 97 pgs: 97 active+clean; 128 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 158 op/s 2024-04-25T19:00:05.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 1206 74.93 MB/sec warmup 3 sec latency 2623.256 ms 2024-04-25T19:00:05.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:05 smithi070 ceph-mon[32820]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T19:00:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:05 smithi114 ceph-mon[32988]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T19:00:05.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:05 smithi063 ceph-mon[28006]: osdmap e114: 12 total, 12 up, 12 in 2024-04-25T19:00:06.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 1890 61.22 MB/sec warmup 4 sec latency 3623.367 ms 2024-04-25T19:00:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:06 smithi070 ceph-mon[32820]: pgmap v426: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:06 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:06 smithi114 ceph-mon[32988]: pgmap v426: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:06 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:06 smithi063 ceph-mon[28006]: pgmap v426: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:06 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:00:07.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 2367 53.85 MB/sec warmup 5 sec latency 4623.477 ms 2024-04-25T19:00:08.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 3030 53.96 MB/sec warmup 6 sec latency 5623.598 ms 2024-04-25T19:00:08.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:08 smithi070 ceph-mon[32820]: pgmap v427: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:08 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:08 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:08 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:00:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:08 smithi114 ceph-mon[32988]: pgmap v427: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:08 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:08 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:08 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:00:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:08 smithi063 ceph-mon[28006]: pgmap v427: 97 pgs: 97 active+clean; 134 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 243 op/s 2024-04-25T19:00:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:08 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:00:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:08 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:00:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:08 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:00:09.069 INFO:tasks.workunit.client.0.smithi063.stdout: 10 3427 51.87 MB/sec warmup 7 sec latency 6623.714 ms 2024-04-25T19:00:10.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 3593 45.72 MB/sec warmup 8 sec latency 7624.010 ms 2024-04-25T19:00:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.188+0000 7f83c5400640 1 -- 172.21.15.63:0/292024198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0102f10 msgr2=0x7f83c01032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.188+0000 7f83c5400640 1 --2- 172.21.15.63:0/292024198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0102f10 0x7f83c01032f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009900 tx=0x7f83b002a480 comp rx=0 tx=0).stop 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 -- 172.21.15.63:0/292024198 shutdown_connections 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 --2- 172.21.15.63:0/292024198 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 0x7f83c010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 --2- 172.21.15.63:0/292024198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0103830 0x7f83c010bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 --2- 172.21.15.63:0/292024198 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0102f10 0x7f83c01032f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 -- 172.21.15.63:0/292024198 >> 172.21.15.63:0/292024198 conn(0x7f83c00fc770 msgr2=0x7f83c00feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 -- 172.21.15.63:0/292024198 shutdown_connections 2024-04-25T19:00:10.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.189+0000 7f83c5400640 1 -- 172.21.15.63:0/292024198 wait complete. 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 Processor -- start 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 -- start start 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 0x7f83c01a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 0x7f83c019a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83c0112010 con 0x7f83c0102f10 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c0111e90 con 0x7f83c0103830 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.190+0000 7f83c5400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83c0112190 con 0x7f83c010c330 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57304/0 (socket says 172.21.15.63:57304) 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 -- 172.21.15.63:0/1207160961 learned_addr learned my addr 172.21.15.63:0/1207160961 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 0x7f83c01a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 0x7f83c019a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 -- 172.21.15.63:0/1207160961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 msgr2=0x7f83c019a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 0x7f83c019a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 -- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 msgr2=0x7f83c01a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 0x7f83c01a03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 -- 172.21.15.63:0/1207160961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c019ad10 con 0x7f83c0103830 2024-04-25T19:00:10.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 0x7f83c01a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.191+0000 7f83bf400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e9f0 tx=0x7f83ac00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.192+0000 7f83bcc00640 1 -- 172.21.15.63:0/1207160961 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00ccc0 con 0x7f83c0103830 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.192+0000 7f83bcc00640 1 -- 172.21.15.63:0/1207160961 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac0108e0 con 0x7f83c0103830 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.192+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c019b000 con 0x7f83c0103830 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.192+0000 7f83bcc00640 1 -- 172.21.15.63:0/1207160961 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac018880 con 0x7f83c0103830 2024-04-25T19:00:10.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.192+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c01a72a0 con 0x7f83c0103830 2024-04-25T19:00:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 Processor -- start 2024-04-25T19:00:10.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 -- start start 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f09b0 0x55825d0f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d0fdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55825d056900 con 0x55825d0fbb90 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55825cffcef0 con 0x55825d0f12d0 2024-04-25T19:00:10.198 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.195+0000 7f83c99bc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55825d0528c0 con 0x55825d0f09b0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57312/0 (socket says 172.21.15.63:57312) 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 -- 172.21.15.63:0/1654521473 learned_addr learned my addr 172.21.15.63:0/1654521473 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bcc00640 1 -- 172.21.15.63:0/1207160961 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f83ac010430 con 0x7f83c0103830 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f09b0 0x55825d0f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 -- 172.21.15.63:0/1654521473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f09b0 msgr2=0x55825d0f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f09b0 0x55825d0f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 -- 172.21.15.63:0/1654521473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 msgr2=0x55825d0fdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d0fdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 -- 172.21.15.63:0/1654521473 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825cd69210 con 0x55825d0f12d0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.196+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f83b0031440 tx=0x7f83b00042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b45acab497fbbbdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83b5200640 1 -- 172.21.15.63:0/1654521473 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0035030 con 0x55825d0f12d0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83b5200640 1 -- 172.21.15.63:0/1654521473 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b00044d0 con 0x55825d0f12d0 2024-04-25T19:00:10.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83b5200640 1 -- 172.21.15.63:0/1654521473 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b00035f0 con 0x55825d0f12d0 2024-04-25T19:00:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83bcc00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 0x7f83a40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83c99bc740 1 -- 172.21.15.63:0/1654521473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 msgr2=0x55825d0fb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.197+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f83b0031440 tx=0x7f83b00042e0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83bcc00640 1 -- 172.21.15.63:0/1207160961 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8631+0+0 (secure 0 0 0) 0x7f83ac0d0c20 con 0x7f83c0103830 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 0x7f83a40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 -- 172.21.15.63:0/1654521473 shutdown_connections 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0fb650 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d0fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1654521473 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f09b0 0x55825d0f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 -- 172.21.15.63:0/1654521473 >> 172.21.15.63:0/1654521473 conn(0x55825d0ecec0 msgr2=0x55825d0ed2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 -- 172.21.15.63:0/1654521473 shutdown_connections 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.198+0000 7f83c99bc740 1 -- 172.21.15.63:0/1654521473 wait complete. 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.199+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 0x7f83a40a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005fd0 tx=0x7f83b8009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.199+0000 7f83c99bc740 1 -- start start 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.199+0000 7f83c99bc740 1 Processor -- start 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d0f8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d0f9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55825d0fe760 con 0x55825d0f09b0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55825d010310 con 0x55825d0fbb90 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55825d0fe8e0 con 0x55825d0f12d0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c99bc740 10 client.0 _ll_register_callbacks cb 0x55825cfdecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83bf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d0f9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57324/0 (socket says 172.21.15.63:57324) 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 -- 172.21.15.63:0/1995357307 learned_addr learned my addr 172.21.15.63:0/1995357307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d0f8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 -- 172.21.15.63:0/1995357307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 msgr2=0x55825d0f9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d0f9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 -- 172.21.15.63:0/1995357307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 msgr2=0x55825d0f8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d0f8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.200+0000 7f83c4a00640 1 -- 172.21.15.63:0/1995357307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55825ce35620 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f83b8003fd0 tx=0x7f83b8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59662dd521261ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83a9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b802c060 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8631+0+0 (secure 0 0 0) 0x7f83b8017030 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b00044d0 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f83b801f840 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83c99bc740 10 client.26798 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55825d10aab0 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.201+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b00044d0 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f83b8027b40 con 0x55825d0fbb90 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 10 client.26798 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 10 client.26798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 20 client.26798 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:10.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d10bf20 0x55825d10e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 4 client.26798 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:00:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55825d06a8b0 con 0x55825d10bf20 2024-04-25T19:00:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.202+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d10bf20 0x55825d10e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.203+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d10bf20 0x55825d10e330 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd6001fbfec7e9ff server_cookie=69fb5a25c1caf44b in_seq=0 out_seq=0 2024-04-25T19:00:10.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.203+0000 7f83a9600640 10 client.26798 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.206+0000 7f83a9600640 1 -- 172.21.15.63:0/1995357307 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55825d06a8b0 con 0x55825d10bf20 2024-04-25T19:00:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.206+0000 7f83a9600640 10 client.26798 handle_command_reply: tid=0 2024-04-25T19:00:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 client.26798 shutdown 2024-04-25T19:00:10.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 msgr2=0x55825d0f9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f83b8003fd0 tx=0x7f83b8004ab0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 shutdown_connections 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d10bf20 0x55825d10e330 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0fbb90 0x55825d0f9640 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83a9600640 0 client.26798 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d0f9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1995357307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d0f8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 >> 172.21.15.63:0/1995357307 conn(0x55825d0ecec0 msgr2=0x55825d0f2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 shutdown_connections 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 1 -- 172.21.15.63:0/1995357307 wait complete. 2024-04-25T19:00:10.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.207+0000 7f83c99bc740 20 client.26798 trim_cache size 0 max 16384 2024-04-25T19:00:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.224+0000 7f83c99bc740 1 Processor -- start 2024-04-25T19:00:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.224+0000 7f83c99bc740 1 -- start start 2024-04-25T19:00:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d107620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0fbb90 0x55825d113c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.227 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01b2720 con 0x55825d0f09b0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01aff30 con 0x55825d0f12d0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c99bc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01adda0 con 0x55825d0fbb90 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57334/0 (socket says 172.21.15.63:57334) 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 -- 172.21.15.63:0/3525812319 learned_addr learned my addr 172.21.15.63:0/3525812319 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bfe00640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d107620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83c4a00640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0fbb90 0x55825d113c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 -- 172.21.15.63:0/3525812319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0fbb90 msgr2=0x55825d113c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0fbb90 0x55825d113c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 -- 172.21.15.63:0/3525812319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 msgr2=0x55825d107620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d107620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bf400640 1 -- 172.21.15.63:0/3525812319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825d0ecce0 con 0x55825d0f12d0 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.225+0000 7f83bfe00640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:10.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.226+0000 7f83bf400640 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83ac098f50 tx=0x7f83ac098d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddc93926946cee43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.226+0000 7f83abe00640 1 -- 172.21.15.63:0/3525812319 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0b6070 con 0x55825d0f12d0 2024-04-25T19:00:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.226+0000 7f83abe00640 1 -- 172.21.15.63:0/3525812319 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac0a0ea0 con 0x55825d0f12d0 2024-04-25T19:00:10.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.226+0000 7f83abe00640 1 -- 172.21.15.63:0/3525812319 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0ce520 con 0x55825d0f12d0 2024-04-25T19:00:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- 172.21.15.63:0/3525812319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 msgr2=0x55825d0ec7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.232 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83ac098f50 tx=0x7f83ac098d90 comp rx=0 tx=0).stop 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- 172.21.15.63:0/3525812319 shutdown_connections 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f12d0 0x55825d0ec7a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0fbb90 0x55825d113c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 --2- 172.21.15.63:0/3525812319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0f09b0 0x55825d107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- 172.21.15.63:0/3525812319 >> 172.21.15.63:0/3525812319 conn(0x55825d0ecec0 msgr2=0x55825d0f7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- 172.21.15.63:0/3525812319 shutdown_connections 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- 172.21.15.63:0/3525812319 wait complete. 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.227+0000 7f83c99bc740 1 -- start start 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 Processor -- start 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d100f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d101450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01ae3f0 con 0x55825d0fbb90 2024-04-25T19:00:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01af860 con 0x55825d0f09b0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01aeac0 con 0x55825d0f12d0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c99bc740 10 client.0 _ll_register_callbacks cb 0x7f83b8003e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83bfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83bf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d100f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83bfe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57336/0 (socket says 172.21.15.63:57336) 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83bfe00640 1 -- 172.21.15.63:0/1639776299 learned_addr learned my addr 172.21.15.63:0/1639776299 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.228+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d101450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 -- 172.21.15.63:0/1639776299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 msgr2=0x55825d100f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d100f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 -- 172.21.15.63:0/1639776299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 msgr2=0x55825d101450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d101450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 -- 172.21.15.63:0/1639776299 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55825d101990 con 0x55825d0f09b0 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83c4a00640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:00:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83bfe00640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009900 tx=0x7f83b0004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e0acab234d84f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83a9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b0035040 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 8631+0+0 (secure 0 0 0) 0x7f83b0052360 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.229+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83ac0a0ea0 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f83b0062050 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83c99bc740 10 client.26804 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55825d1720f0 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83ac0a0ea0 con 0x55825d0f09b0 2024-04-25T19:00:10.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f83b003b430 con 0x55825d0f09b0 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83c99bc740 10 client.26804 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83c99bc740 10 client.26804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.230+0000 7f83c99bc740 20 client.26804 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d172430 0x55825d174850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83c99bc740 4 client.26804 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55825d064320 con 0x55825d172430 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83bf400640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d172430 0x55825d174850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83bf400640 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d172430 0x55825d174850 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=777c076281a34cc7 server_cookie=b94ee3f96f795de4 in_seq=0 out_seq=0 2024-04-25T19:00:10.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.231+0000 7f83a9600640 10 client.26804 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:10.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:10 smithi070 ceph-mon[32820]: pgmap v428: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.731+0000 7f83a9600640 1 -- 172.21.15.63:0/1639776299 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55825d064320 con 0x55825d172430 2024-04-25T19:00:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83a9600640 10 client.26804 handle_command_reply: tid=0 2024-04-25T19:00:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 client.26804 shutdown 2024-04-25T19:00:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 msgr2=0x55825d103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009900 tx=0x7f83b0004540 comp rx=0 tx=0).stop 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 shutdown_connections 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55825d172430 0x55825d174850 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55825d0fbb90 0x55825d101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55825d0f12d0 0x55825d100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 --2- 172.21.15.63:0/1639776299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55825d0f09b0 0x55825d103ba0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83a9600640 0 client.26804 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.732+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 >> 172.21.15.63:0/1639776299 conn(0x55825d0ecec0 msgr2=0x55825d0f2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.733+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 shutdown_connections 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.733+0000 7f83c99bc740 1 -- 172.21.15.63:0/1639776299 wait complete. 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.733+0000 7f83c99bc740 20 client.26804 trim_cache size 0 max 16384 2024-04-25T19:00:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:00:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:00:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:00:10.736 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:00:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.736+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 msgr2=0x7f83a40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.736+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 0x7f83a40a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005fd0 tx=0x7f83b8009500 comp rx=0 tx=0).stop 2024-04-25T19:00:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.736+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 msgr2=0x7f83c01a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:10.737 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.736+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00e9f0 tx=0x7f83ac00eeb0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 shutdown_connections 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f83a40a3d10 0x7f83a40a61d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f83c0103830 0x7f83c01a0920 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f83c010c330 0x7f83c019a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 --2- 172.21.15.63:0/1207160961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83c0102f10 0x7f83c01a03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:10.738 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 >> 172.21.15.63:0/1207160961 conn(0x7f83c00fc770 msgr2=0x7f83c0100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 shutdown_connections 2024-04-25T19:00:10.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:10.737+0000 7f83c5400640 1 -- 172.21.15.63:0/1207160961 wait complete. 2024-04-25T19:00:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:10 smithi114 ceph-mon[32988]: pgmap v428: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:10 smithi063 ceph-mon[28006]: pgmap v428: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:11.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 3736 40.70 MB/sec warmup 9 sec latency 8624.110 ms 2024-04-25T19:00:12.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 3975 37.20 MB/sec warmup 10 sec latency 9624.193 ms 2024-04-25T19:00:12.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:12 smithi070 ceph-mon[32820]: pgmap v429: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:12 smithi114 ceph-mon[32988]: pgmap v429: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:12 smithi063 ceph-mon[28006]: pgmap v429: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:13.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 4428 36.06 MB/sec warmup 11 sec latency 10624.297 ms 2024-04-25T19:00:14.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 5183 34.92 MB/sec warmup 12 sec latency 11624.428 ms 2024-04-25T19:00:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:14 smithi070 ceph-mon[32820]: pgmap v430: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:14.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:14 smithi070 ceph-mon[32820]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T19:00:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:14 smithi114 ceph-mon[32988]: pgmap v430: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:14 smithi114 ceph-mon[32988]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T19:00:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:14 smithi063 ceph-mon[28006]: pgmap v430: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 229 op/s 2024-04-25T19:00:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:14 smithi063 ceph-mon[28006]: osdmap e115: 12 total, 12 up, 12 in 2024-04-25T19:00:15.070 INFO:tasks.workunit.client.0.smithi063.stdout: 10 5707 35.65 MB/sec warmup 13 sec latency 12624.527 ms 2024-04-25T19:00:16.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 6308 36.11 MB/sec warmup 14 sec latency 13624.623 ms 2024-04-25T19:00:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:16 smithi070 ceph-mon[32820]: pgmap v432: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:16 smithi114 ceph-mon[32988]: pgmap v432: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:16 smithi063 ceph-mon[28006]: pgmap v432: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:17.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 6634 35.71 MB/sec warmup 15 sec latency 13767.630 ms 2024-04-25T19:00:18.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 6783 33.63 MB/sec warmup 16 sec latency 1855.724 ms 2024-04-25T19:00:18.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:18 smithi070 ceph-mon[32820]: pgmap v433: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:18 smithi114 ceph-mon[32988]: pgmap v433: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:18 smithi063 ceph-mon[28006]: pgmap v433: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 550 op/s 2024-04-25T19:00:19.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 6939 31.68 MB/sec warmup 17 sec latency 2855.904 ms 2024-04-25T19:00:20.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 7416 31.15 MB/sec warmup 18 sec latency 3171.811 ms 2024-04-25T19:00:20.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:20 smithi070 ceph-mon[32820]: pgmap v434: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:20 smithi114 ceph-mon[32988]: pgmap v434: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:20.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:20 smithi063 ceph-mon[28006]: pgmap v434: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:21.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 8629 32.44 MB/sec warmup 19 sec latency 58.880 ms 2024-04-25T19:00:22.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 9732 35.55 MB/sec warmup 20 sec latency 42.537 ms 2024-04-25T19:00:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:22 smithi070 ceph-mon[32820]: pgmap v435: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:22 smithi114 ceph-mon[32988]: pgmap v435: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:22.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:22 smithi063 ceph-mon[28006]: pgmap v435: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:23.071 INFO:tasks.workunit.client.0.smithi063.stdout: 10 10143 34.80 MB/sec warmup 21 sec latency 87.766 ms 2024-04-25T19:00:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 -- 172.21.15.63:0/237976715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806b3a0 msgr2=0x7f7708106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:23.924 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 --2- 172.21.15.63:0/237976715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806b3a0 0x7f7708106b80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009900 tx=0x7f76f402a410 comp rx=0 tx=0).stop 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 -- 172.21.15.63:0/237976715 shutdown_connections 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 --2- 172.21.15.63:0/237976715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806b3a0 0x7f7708106b80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 --2- 172.21.15.63:0/237976715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7708107260 0x7f770810ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 --2- 172.21.15.63:0/237976715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f770806a9f0 0x7f770806add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.923+0000 7f7710200640 1 -- 172.21.15.63:0/237976715 >> 172.21.15.63:0/237976715 conn(0x7f7708076c50 msgr2=0x7f7708077060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.924+0000 7f7710200640 1 -- 172.21.15.63:0/237976715 shutdown_connections 2024-04-25T19:00:23.925 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.924+0000 7f7710200640 1 -- 172.21.15.63:0/237976715 wait complete. 2024-04-25T19:00:23.926 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.925+0000 7f7710200640 1 Processor -- start 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.925+0000 7f7710200640 1 -- start start 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 0x7f770810ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770810bfa0 0x7f770810c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77081028d0 con 0x7f7708107260 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7708102750 con 0x7f770806a9f0 2024-04-25T19:00:23.927 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f7710200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7708102a50 con 0x7f770810bfa0 2024-04-25T19:00:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f770ce00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 0x7f770810ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f770d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.928 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f770d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55700/0 (socket says 172.21.15.63:55700) 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.926+0000 7f770d800640 1 -- 172.21.15.63:0/2047493936 learned_addr learned my addr 172.21.15.63:0/2047493936 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770ce00640 1 -- 172.21.15.63:0/2047493936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770810bfa0 msgr2=0x7f770810c450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770ce00640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770810bfa0 0x7f770810c450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770ce00640 1 -- 172.21.15.63:0/2047493936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 msgr2=0x7f770810e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770ce00640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770ce00640 1 -- 172.21.15.63:0/2047493936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77081ab480 con 0x7f7708107260 2024-04-25T19:00:23.929 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.927+0000 7f770d800640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.928+0000 7f770ce00640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 0x7f770810ba60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009850 tx=0x7f76f4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.928+0000 7f7706000640 1 -- 172.21.15.63:0/2047493936 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f403f070 con 0x7f7708107260 2024-04-25T19:00:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.928+0000 7f7706000640 1 -- 172.21.15.63:0/2047493936 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f4031b50 con 0x7f7708107260 2024-04-25T19:00:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.928+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77081ab650 con 0x7f7708107260 2024-04-25T19:00:23.930 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.928+0000 7f7706000640 1 -- 172.21.15.63:0/2047493936 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f4035030 con 0x7f7708107260 2024-04-25T19:00:23.931 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.929+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77081abb60 con 0x7f7708107260 2024-04-25T19:00:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.933+0000 7f7712f71740 1 Processor -- start 2024-04-25T19:00:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 -- start start 2024-04-25T19:00:23.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8b91f900 con 0x560c8b9ba140 2024-04-25T19:00:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560c8b8c5ef0 con 0x560c8b9b9820 2024-04-25T19:00:23.936 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.934+0000 7f7712f71740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560c8b91b8c0 con 0x560c8b9c4a00 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f7706000640 1 -- 172.21.15.63:0/2047493936 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f76f4042070 con 0x7f7708107260 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55716/0 (socket says 172.21.15.63:55716) 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 -- 172.21.15.63:0/3103113025 learned_addr learned my addr 172.21.15.63:0/3103113025 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770ce00640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770d800640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 -- 172.21.15.63:0/3103113025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 msgr2=0x560c8b9c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.935+0000 7f770e200640 1 -- 172.21.15.63:0/3103113025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 msgr2=0x560c8b9c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.936+0000 7f770e200640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.936+0000 7f770e200640 1 -- 172.21.15.63:0/3103113025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8b632210 con 0x560c8b9b9820 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.936+0000 7f770d800640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.936+0000 7f770e200640 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7700009db0 tx=0x7f770000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f9cf696958eff24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.937+0000 7f76faa00640 1 -- 172.21.15.63:0/3103113025 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700004b20 con 0x560c8b9b9820 2024-04-25T19:00:23.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.937+0000 7f76faa00640 1 -- 172.21.15.63:0/3103113025 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7700004cc0 con 0x560c8b9b9820 2024-04-25T19:00:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.937+0000 7f76faa00640 1 -- 172.21.15.63:0/3103113025 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77000187f0 con 0x560c8b9b9820 2024-04-25T19:00:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.937+0000 7f7706000640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 0x7f76f00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.937+0000 7f7706000640 1 -- 172.21.15.63:0/2047493936 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f76f40f06a0 con 0x7f7708107260 2024-04-25T19:00:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.938+0000 7f7712f71740 1 -- 172.21.15.63:0/3103113025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 msgr2=0x560c8b9b9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:23.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.938+0000 7f7712f71740 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7700009db0 tx=0x7f770000e720 comp rx=0 tx=0).stop 2024-04-25T19:00:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.939+0000 7f770d800640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 0x7f76f00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.939+0000 7f7712f71740 1 -- 172.21.15.63:0/3103113025 shutdown_connections 2024-04-25T19:00:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.939+0000 7f7712f71740 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.939+0000 7f770d800640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 0x7f76f00a6380 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009620 tx=0x7f76fc005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.939+0000 7f7712f71740 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.940+0000 7f7712f71740 1 --2- 172.21.15.63:0/3103113025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9b9c00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.940+0000 7f7712f71740 1 -- 172.21.15.63:0/3103113025 >> 172.21.15.63:0/3103113025 conn(0x560c8b9b5d30 msgr2=0x560c8b9b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.940+0000 7f7712f71740 1 -- 172.21.15.63:0/3103113025 shutdown_connections 2024-04-25T19:00:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.940+0000 7f7712f71740 1 -- 172.21.15.63:0/3103113025 wait complete. 2024-04-25T19:00:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 -- start start 2024-04-25T19:00:23.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 Processor -- start 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 0x560c8b9c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8b9c7630 con 0x560c8b9ba140 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x560c8b8d9310 con 0x560c8b9c4a00 2024-04-25T19:00:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.941+0000 7f7712f71740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x560c8b9c77b0 con 0x560c8b9b9820 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f7712f71740 10 client.0 _ll_register_callbacks cb 0x560c8b8a7cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770d800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 0x560c8b9c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55728/0 (socket says 172.21.15.63:55728) 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770e200640 1 -- 172.21.15.63:0/3808544951 learned_addr learned my addr 172.21.15.63:0/3808544951 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:23.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770ce00640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.942+0000 7f770e200640 1 -- 172.21.15.63:0/3808544951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 msgr2=0x560c8b9c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770e200640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 0x560c8b9c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770e200640 1 -- 172.21.15.63:0/3808544951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 msgr2=0x560c8b9c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770e200640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770e200640 1 -- 172.21.15.63:0/3808544951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8b6fe620 con 0x560c8b9c4a00 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770ce00640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770e200640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7700005c40 tx=0x7f7700005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0978c043877db32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f76ef400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f770d800640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 0x560c8b9c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700009a50 con 0x560c8b9c4a00 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7700004cc0 con 0x560c8b9c4a00 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.943+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f7700003970 con 0x560c8b9c4a00 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.944+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7700003b70 con 0x560c8b9c4a00 2024-04-25T19:00:23.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.944+0000 7f7712f71740 10 client.26816 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.944+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x560c8b9d3920 con 0x560c8b9c4a00 2024-04-25T19:00:23.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.944+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7700004cc0 con 0x560c8b9c4a00 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f7700033020 con 0x560c8b9c4a00 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 10 client.26816 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 10 client.26816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 20 client.26816 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8b9d4d30 0x560c8b9d7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 4 client.26816 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.945+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c8b9338b0 con 0x560c8b9d4d30 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.946+0000 7f770d800640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8b9d4d30 0x560c8b9d7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.946+0000 7f770d800640 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8b9d4d30 0x560c8b9d7120 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42115077752f71c3 server_cookie=6b4927e37290a88e in_seq=0 out_seq=0 2024-04-25T19:00:23.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.946+0000 7f76ef400640 10 client.26816 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:23.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.951+0000 7f76ef400640 1 -- 172.21.15.63:0/3808544951 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560c8b9338b0 con 0x560c8b9d4d30 2024-04-25T19:00:23.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.951+0000 7f76ef400640 10 client.26816 handle_command_reply: tid=0 2024-04-25T19:00:23.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.951+0000 7f7712f71740 1 client.26816 shutdown 2024-04-25T19:00:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 msgr2=0x560c8b9c2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7700005c40 tx=0x7f7700005740 comp rx=0 tx=0).stop 2024-04-25T19:00:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 shutdown_connections 2024-04-25T19:00:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8b9d4d30 0x560c8b9d7120 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9c2490 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 --2- 172.21.15.63:0/3808544951 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9b9820 0x560c8b9c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 >> 172.21.15.63:0/3808544951 conn(0x560c8b9b5d30 msgr2=0x560c8b9bbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f76ef400640 0 client.26816 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.952+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 shutdown_connections 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.953+0000 7f7712f71740 1 -- 172.21.15.63:0/3808544951 wait complete. 2024-04-25T19:00:23.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.953+0000 7f7712f71740 20 client.26816 trim_cache size 0 max 16384 2024-04-25T19:00:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.977+0000 7f7712f71740 1 Processor -- start 2024-04-25T19:00:23.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.977+0000 7f7712f71740 1 -- start start 2024-04-25T19:00:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9b9820 0x560c8b9d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9ba140 0x560c8b9c0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77081ae3a0 con 0x560c8b9b9820 2024-04-25T19:00:23.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f770811d460 con 0x560c8b9c4a00 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f7712f71740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f77081036a0 con 0x560c8b9ba140 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f770ce00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9ba140 0x560c8b9c0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f770d800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9b9820 0x560c8b9d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f770e200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55740/0 (socket says 172.21.15.63:55740) 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.978+0000 7f770e200640 1 -- 172.21.15.63:0/1085305555 learned_addr learned my addr 172.21.15.63:0/1085305555 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:23.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 -- 172.21.15.63:0/1085305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9ba140 msgr2=0x560c8b9c0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9ba140 0x560c8b9c0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 -- 172.21.15.63:0/1085305555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9b9820 msgr2=0x560c8b9d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9b9820 0x560c8b9d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 -- 172.21.15.63:0/1085305555 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8b9bd5f0 con 0x560c8b9c4a00 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f770e200640 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7700006010 tx=0x7f7700009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3714b22d032943ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.979+0000 7f76fa000640 1 -- 172.21.15.63:0/1085305555 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700041070 con 0x560c8b9c4a00 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.980+0000 7f76fa000640 1 -- 172.21.15.63:0/1085305555 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7700004c50 con 0x560c8b9c4a00 2024-04-25T19:00:23.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.980+0000 7f76fa000640 1 -- 172.21.15.63:0/1085305555 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77000044f0 con 0x560c8b9c4a00 2024-04-25T19:00:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- 172.21.15.63:0/1085305555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 msgr2=0x560c8b9dc6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7700006010 tx=0x7f7700009500 comp rx=0 tx=0).stop 2024-04-25T19:00:23.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- 172.21.15.63:0/1085305555 shutdown_connections 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9c4a00 0x560c8b9dc6f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9ba140 0x560c8b9c0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 --2- 172.21.15.63:0/1085305555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9b9820 0x560c8b9d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- 172.21.15.63:0/1085305555 >> 172.21.15.63:0/1085305555 conn(0x560c8b9b5d30 msgr2=0x560c8b9b0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- 172.21.15.63:0/1085305555 shutdown_connections 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- 172.21.15.63:0/1085305555 wait complete. 2024-04-25T19:00:23.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.981+0000 7f7712f71740 1 -- start start 2024-04-25T19:00:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 Processor -- start 2024-04-25T19:00:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9ca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f770811b960 con 0x560c8b9ba140 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f770811cd90 con 0x560c8b9b9820 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f770811bff0 con 0x560c8b9c4a00 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.982+0000 7f7712f71740 10 client.0 _ll_register_callbacks cb 0x7f7700009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:55746/0 (socket says 172.21.15.63:55746) 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770e200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39502/0 (socket says 172.21.15.63:39502) 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 -- 172.21.15.63:0/60467065 learned_addr learned my addr 172.21.15.63:0/60467065 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770ce00640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 -- 172.21.15.63:0/60467065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 msgr2=0x560c8b9ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 -- 172.21.15.63:0/60467065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 msgr2=0x560c8b9c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:23.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:23.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.983+0000 7f770d800640 1 -- 172.21.15.63:0/60467065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8ba38fb0 con 0x560c8b9b9820 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.984+0000 7f770d800640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x560c8b9d7690 tx=0x7f76fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecbea5bdfe299ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.984+0000 7f76ef400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.984+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc02a380 con 0x560c8b9b9820 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.984+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7700004c50 con 0x560c8b9b9820 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.984+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f76fc02a520 con 0x560c8b9b9820 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.985+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f76fc02a720 con 0x560c8b9b9820 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.985+0000 7f7712f71740 10 client.26822 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:23.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.985+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x560c8ba3a2c0 con 0x560c8b9b9820 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.985+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7700004c50 con 0x560c8b9b9820 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.985+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f76fc033470 con 0x560c8b9b9820 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 10 client.26822 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 10 client.26822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 20 client.26822 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8ba3ab20 0x560c8ba3cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 4 client.26822 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c8b92d320 con 0x560c8ba3ab20 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f770ce00640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8ba3ab20 0x560c8ba3cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:23.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f770ce00640 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8ba3ab20 0x560c8ba3cf40 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35d59949dd2fe7e9 server_cookie=ae4c2b54dc33c58c in_seq=0 out_seq=0 2024-04-25T19:00:23.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:23.986+0000 7f76ef400640 10 client.26822 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:24.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 10665 33.66 MB/sec warmup 22 sec latency 103.402 ms 2024-04-25T19:00:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.290+0000 7f76ef400640 1 -- 172.21.15.63:0/60467065 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c8b92d320 con 0x560c8ba3ab20 2024-04-25T19:00:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.290+0000 7f76ef400640 10 client.26822 handle_command_reply: tid=0 2024-04-25T19:00:24.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.290+0000 7f7712f71740 1 client.26822 shutdown 2024-04-25T19:00:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 msgr2=0x560c8b9c99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x560c8b9d7690 tx=0x7f76fc004730 comp rx=0 tx=0).stop 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 shutdown_connections 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x560c8ba3ab20 0x560c8ba3cf40 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x560c8b9c4a00 0x560c8b9ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8b9ba140 0x560c8b9c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 --2- 172.21.15.63:0/60467065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x560c8b9b9820 0x560c8b9c99d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f76ef400640 0 client.26822 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.291+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 >> 172.21.15.63:0/60467065 conn(0x560c8b9b5d30 msgr2=0x560c8b9c0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.292+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 shutdown_connections 2024-04-25T19:00:24.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.292+0000 7f7712f71740 1 -- 172.21.15.63:0/60467065 wait complete. 2024-04-25T19:00:24.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.292+0000 7f7712f71740 20 client.26822 trim_cache size 0 max 16384 2024-04-25T19:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:00:24.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.294+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 msgr2=0x7f76f00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.294+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 0x7f76f00a6380 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009620 tx=0x7f76fc005f90 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.294+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 msgr2=0x7f770810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.294+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 0x7f770810ba60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009850 tx=0x7f76f4003b90 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.294+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 shutdown_connections 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.295+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f76f00a3ec0 0x7f76f00a6380 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.295+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f770810bfa0 0x7f770810c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.295+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7708107260 0x7f770810ba60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.295+0000 7f7710200640 1 --2- 172.21.15.63:0/2047493936 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f770806a9f0 0x7f770810e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.295+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 >> 172.21.15.63:0/2047493936 conn(0x7f7708076c50 msgr2=0x7f77080fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.296+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 shutdown_connections 2024-04-25T19:00:24.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:24.296+0000 7f7710200640 1 -- 172.21.15.63:0/2047493936 wait complete. 2024-04-25T19:00:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:24 smithi070 ceph-mon[32820]: pgmap v436: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:24 smithi114 ceph-mon[32988]: pgmap v436: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:24 smithi063 ceph-mon[28006]: pgmap v436: 97 pgs: 97 active+clean; 152 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.2 MiB/s wr, 549 op/s 2024-04-25T19:00:25.072 INFO:tasks.workunit.client.0.smithi063.stdout: 10 11400 33.98 MB/sec warmup 23 sec latency 163.723 ms 2024-04-25T19:00:26.072 INFO:tasks.workunit.client.0.smithi063.stdout: 10 12676 35.67 MB/sec warmup 24 sec latency 52.275 ms 2024-04-25T19:00:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:26 smithi114 ceph-mon[32988]: pgmap v437: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 465 op/s 2024-04-25T19:00:26.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:26 smithi063 ceph-mon[28006]: pgmap v437: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 465 op/s 2024-04-25T19:00:26.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:26 smithi070 ceph-mon[32820]: pgmap v437: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.5 MiB/s wr, 465 op/s 2024-04-25T19:00:27.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 13452 36.60 MB/sec warmup 25 sec latency 35.158 ms 2024-04-25T19:00:28.072 INFO:tasks.workunit.client.0.smithi063.stdout: 10 13845 35.41 MB/sec warmup 26 sec latency 81.626 ms 2024-04-25T19:00:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:28 smithi114 ceph-mon[32988]: pgmap v438: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 317 op/s 2024-04-25T19:00:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:28 smithi063 ceph-mon[28006]: pgmap v438: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 317 op/s 2024-04-25T19:00:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:28 smithi070 ceph-mon[32820]: pgmap v438: 97 pgs: 97 active+clean; 174 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 317 op/s 2024-04-25T19:00:29.072 INFO:tasks.workunit.client.0.smithi063.stdout: 10 14798 35.67 MB/sec warmup 27 sec latency 58.277 ms 2024-04-25T19:00:30.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 15988 37.03 MB/sec warmup 28 sec latency 40.406 ms 2024-04-25T19:00:30.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:30 smithi063 ceph-mon[28006]: pgmap v439: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-04-25T19:00:30.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:30 smithi070 ceph-mon[32820]: pgmap v439: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-04-25T19:00:30.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:30 smithi114 ceph-mon[32988]: pgmap v439: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 667 op/s 2024-04-25T19:00:31.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 16777 37.59 MB/sec warmup 29 sec latency 44.370 ms 2024-04-25T19:00:32.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 17253 36.72 MB/sec warmup 30 sec latency 102.980 ms 2024-04-25T19:00:32.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:32 smithi063 ceph-mon[28006]: pgmap v440: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-04-25T19:00:32.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:32 smithi070 ceph-mon[32820]: pgmap v440: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-04-25T19:00:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:32 smithi114 ceph-mon[32988]: pgmap v440: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-04-25T19:00:33.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 17858 36.56 MB/sec warmup 31 sec latency 158.653 ms 2024-04-25T19:00:34.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 19092 37.25 MB/sec warmup 32 sec latency 31.526 ms 2024-04-25T19:00:34.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:34 smithi063 ceph-mon[28006]: pgmap v441: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T19:00:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:34 smithi070 ceph-mon[32820]: pgmap v441: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T19:00:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:34 smithi114 ceph-mon[32988]: pgmap v441: 97 pgs: 97 active+clean; 181 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 591 op/s 2024-04-25T19:00:35.074 INFO:tasks.workunit.client.0.smithi063.stdout: 10 19988 38.25 MB/sec warmup 33 sec latency 98.049 ms 2024-04-25T19:00:36.073 INFO:tasks.workunit.client.0.smithi063.stdout: 10 20447 37.72 MB/sec warmup 34 sec latency 53.166 ms 2024-04-25T19:00:36.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:36 smithi063 ceph-mon[28006]: pgmap v442: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 793 op/s 2024-04-25T19:00:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:36 smithi070 ceph-mon[32820]: pgmap v442: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 793 op/s 2024-04-25T19:00:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:36 smithi114 ceph-mon[32988]: pgmap v442: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 15 MiB/s wr, 793 op/s 2024-04-25T19:00:37.074 INFO:tasks.workunit.client.0.smithi063.stdout: 10 21016 37.25 MB/sec warmup 35 sec latency 74.939 ms 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 -- 172.21.15.63:0/55078598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1092c0 msgr2=0x7ff00c110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 --2- 172.21.15.63:0/55078598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1092c0 0x7ff00c110e90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff004009ca0 tx=0x7ff00402a8c0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 -- 172.21.15.63:0/55078598 shutdown_connections 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 --2- 172.21.15.63:0/55078598 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c104ba0 0x7ff00c108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 --2- 172.21.15.63:0/55078598 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1092c0 0x7ff00c110e90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 --2- 172.21.15.63:0/55078598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c1041f0 0x7ff00c1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.479+0000 7ff00b200640 1 -- 172.21.15.63:0/55078598 >> 172.21.15.63:0/55078598 conn(0x7ff00c0ffa30 msgr2=0x7ff00c101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.481 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.480+0000 7ff00b200640 1 -- 172.21.15.63:0/55078598 shutdown_connections 2024-04-25T19:00:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.480+0000 7ff00b200640 1 -- 172.21.15.63:0/55078598 wait complete. 2024-04-25T19:00:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 Processor -- start 2024-04-25T19:00:37.482 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 -- start start 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 0x7ff00c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 0x7ff00c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c19efc0 0x7ff00c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c114260 con 0x7ff00c19efc0 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1140e0 con 0x7ff00c1092c0 2024-04-25T19:00:37.483 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.481+0000 7ff00b200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1143e0 con 0x7ff00c1041f0 2024-04-25T19:00:37.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff009e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 0x7ff00c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff009e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 0x7ff00c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40812/0 (socket says 172.21.15.63:40812) 2024-04-25T19:00:37.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff009e00640 1 -- 172.21.15.63:0/449610496 learned_addr learned my addr 172.21.15.63:0/449610496 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:37.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 0x7ff00c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.484 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff00a800640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c19efc0 0x7ff00c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 -- 172.21.15.63:0/449610496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 msgr2=0x7ff00c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 0x7ff00c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 -- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c19efc0 msgr2=0x7ff00c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c19efc0 0x7ff00c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.482+0000 7ff003e00640 1 -- 172.21.15.63:0/449610496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00c19fab0 con 0x7ff00c1092c0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff003e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 0x7ff00c19ac40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feff400e9c0 tx=0x7feff400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff002a00640 1 -- 172.21.15.63:0/449610496 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff400ccc0 con 0x7ff00c1092c0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff002a00640 1 -- 172.21.15.63:0/449610496 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feff40108e0 con 0x7ff00c1092c0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00c19fda0 con 0x7ff00c1092c0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00c1a7590 con 0x7ff00c1092c0 2024-04-25T19:00:37.485 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.483+0000 7ff002a00640 1 -- 172.21.15.63:0/449610496 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40187c0 con 0x7ff00c1092c0 2024-04-25T19:00:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.488+0000 7ff01418e740 1 Processor -- start 2024-04-25T19:00:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.488+0000 7ff002a00640 1 -- 172.21.15.63:0/449610496 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7feff4010430 con 0x7ff00c1092c0 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 -- start start 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc4022d0 0x5600dc40c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc40cb90 0x5600dc40ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5600dc367900 con 0x5600dc40cb90 2024-04-25T19:00:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600dc30def0 con 0x5600dc4019b0 2024-04-25T19:00:37.491 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.489+0000 7ff01418e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5600dc3638c0 con 0x5600dc4022d0 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:51532/0 (socket says 172.21.15.63:51532) 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 -- 172.21.15.63:0/3025738874 learned_addr learned my addr 172.21.15.63:0/3025738874 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff009e00640 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc4022d0 0x5600dc40c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 -- 172.21.15.63:0/3025738874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc4022d0 msgr2=0x5600dc40c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc4022d0 0x5600dc40c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 -- 172.21.15.63:0/3025738874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc40cb90 msgr2=0x5600dc40ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.492 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc40cb90 0x5600dc40ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 -- 172.21.15.63:0/3025738874 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600dc07a210 con 0x5600dc4019b0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff002a00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 0x7feff00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.490+0000 7ff00a800640 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff004009ca0 tx=0x7ff00402f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41921a9137d52ac5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.491+0000 7ff002a00640 1 -- 172.21.15.63:0/449610496 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7feff40d0bd0 con 0x7ff00c1092c0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.491+0000 7feffb400640 1 -- 172.21.15.63:0/3025738874 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff004031e30 con 0x5600dc4019b0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.491+0000 7ff009e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 0x7feff00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.491+0000 7feffb400640 1 -- 172.21.15.63:0/3025738874 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff004036030 con 0x5600dc4019b0 2024-04-25T19:00:37.493 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.491+0000 7feffb400640 1 -- 172.21.15.63:0/3025738874 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040043c0 con 0x5600dc4019b0 2024-04-25T19:00:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 -- 172.21.15.63:0/3025738874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 msgr2=0x5600dc401d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff004009ca0 tx=0x7ff00402f3c0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff009e00640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 0x7feff00a61d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5600dc402f50 tx=0x7feffc0084c0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.494 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 -- 172.21.15.63:0/3025738874 shutdown_connections 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc40cb90 0x5600dc40ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc4022d0 0x5600dc40c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 --2- 172.21.15.63:0/3025738874 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc401d90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 -- 172.21.15.63:0/3025738874 >> 172.21.15.63:0/3025738874 conn(0x5600dc3fdec0 msgr2=0x5600dc3fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 -- 172.21.15.63:0/3025738874 shutdown_connections 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.492+0000 7ff01418e740 1 -- 172.21.15.63:0/3025738874 wait complete. 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 -- start start 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 Processor -- start 2024-04-25T19:00:37.495 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc409ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.496 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40cb90 0x5600dc40a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5600dc40f5e0 con 0x5600dc4022d0 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5600dc321310 con 0x5600dc4019b0 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5600dc40f760 con 0x5600dc40cb90 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.493+0000 7ff01418e740 10 client.0 _ll_register_callbacks cb 0x5600dc2efcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60292/0 (socket says 172.21.15.63:60292) 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 -- 172.21.15.63:0/929274614 learned_addr learned my addr 172.21.15.63:0/929274614 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff00a800640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40cb90 0x5600dc40a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 -- 172.21.15.63:0/929274614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40cb90 msgr2=0x5600dc40a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40cb90 0x5600dc40a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 -- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 msgr2=0x5600dc409ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:00:37.497 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc409ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.494+0000 7ff003e00640 1 -- 172.21.15.63:0/929274614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600dc146620 con 0x5600dc4022d0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7ff003e00640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7feff409c7d0 tx=0x7feff40107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea507748e8509896 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7fefefe00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40a0e80 con 0x5600dc4022d0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7feff409add0 con 0x5600dc4022d0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff004036030 con 0x5600dc4022d0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7feff40a5740 con 0x5600dc4022d0 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7ff01418e740 10 client.16635 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:37.498 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.495+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5600dc424190 con 0x5600dc4022d0 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff004036030 con 0x5600dc4022d0 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7feff4098820 con 0x5600dc4022d0 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 10 client.16635 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 20 client.16635 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:37.499 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc425650 0x5600dc427a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 4 client.16635 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.496+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600dc37b8b0 con 0x5600dc425650 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.497+0000 7ff009e00640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc425650 0x5600dc427a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.497+0000 7ff009e00640 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc425650 0x5600dc427a40 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9881715a5434e5 server_cookie=28fcc4453181bef in_seq=0 out_seq=0 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.497+0000 7fefefe00640 10 client.16635 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.498+0000 7fefefe00640 1 -- 172.21.15.63:0/929274614 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600dc37b8b0 con 0x5600dc425650 2024-04-25T19:00:37.500 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.498+0000 7fefefe00640 10 client.16635 handle_command_reply: tid=0 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.498+0000 7ff01418e740 1 client.16635 shutdown 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 msgr2=0x5600dc40a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7feff409c7d0 tx=0x7feff40107e0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 shutdown_connections 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc425650 0x5600dc427a40 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40cb90 0x5600dc40a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4022d0 0x5600dc40a0e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 --2- 172.21.15.63:0/929274614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc4019b0 0x5600dc409ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7fefefe00640 0 client.16635 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.499+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 >> 172.21.15.63:0/929274614 conn(0x5600dc3fdec0 msgr2=0x5600dc403bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.500+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 shutdown_connections 2024-04-25T19:00:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.500+0000 7ff01418e740 1 -- 172.21.15.63:0/929274614 wait complete. 2024-04-25T19:00:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.500+0000 7ff01418e740 20 client.16635 trim_cache size 0 max 16384 2024-04-25T19:00:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 Processor -- start 2024-04-25T19:00:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 -- start start 2024-04-25T19:00:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc40a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc42d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1ae150 con 0x5600dc4019b0 2024-04-25T19:00:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1ab960 con 0x5600dc40cb90 2024-04-25T19:00:37.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.522+0000 7ff01418e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1a97d0 con 0x5600dc40c3e0 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff00a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc42d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff009e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc40a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:51538/0 (socket says 172.21.15.63:51538) 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 -- 172.21.15.63:0/1305507903 learned_addr learned my addr 172.21.15.63:0/1305507903 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 -- 172.21.15.63:0/1305507903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 msgr2=0x5600dc42d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc42d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 -- 172.21.15.63:0/1305507903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 msgr2=0x5600dc40a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc40a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 -- 172.21.15.63:0/1305507903 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600dc428070 con 0x5600dc40cb90 2024-04-25T19:00:37.525 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff009e00640 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc40a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff00a800640 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc42d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.523+0000 7ff003e00640 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feff40a5210 tx=0x7feff400efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e99b34aaf2d47c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.524+0000 7feffaa00640 1 -- 172.21.15.63:0/1305507903 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff409bc80 con 0x5600dc40cb90 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.524+0000 7feffaa00640 1 -- 172.21.15.63:0/1305507903 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feff409add0 con 0x5600dc40cb90 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.524+0000 7feffaa00640 1 -- 172.21.15.63:0/1305507903 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40ce500 con 0x5600dc40cb90 2024-04-25T19:00:37.526 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 -- 172.21.15.63:0/1305507903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 msgr2=0x5600dc404370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feff40a5210 tx=0x7feff400efb0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 -- 172.21.15.63:0/1305507903 shutdown_connections 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc404370 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc42d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 --2- 172.21.15.63:0/1305507903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc40a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 -- 172.21.15.63:0/1305507903 >> 172.21.15.63:0/1305507903 conn(0x5600dc3fdec0 msgr2=0x5600dc3fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 -- 172.21.15.63:0/1305507903 shutdown_connections 2024-04-25T19:00:37.527 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.525+0000 7ff01418e740 1 -- 172.21.15.63:0/1305507903 wait complete. 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.526+0000 7ff01418e740 1 -- start start 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.526+0000 7ff01418e740 1 Processor -- start 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc41ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc41b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1a9e20 con 0x5600dc4019b0 2024-04-25T19:00:37.529 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1ab290 con 0x5600dc40cb90 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1aa4f0 con 0x5600dc40c3e0 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff01418e740 10 client.0 _ll_register_callbacks cb 0x7feff409c6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff00a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc41b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:40852/0 (socket says 172.21.15.63:40852) 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 -- 172.21.15.63:0/1973262306 learned_addr learned my addr 172.21.15.63:0/1973262306 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff009e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc41ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 -- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 msgr2=0x5600dc41b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc41b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 -- 172.21.15.63:0/1973262306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 msgr2=0x5600dc41ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc41ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.530 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.527+0000 7ff003e00640 1 -- 172.21.15.63:0/1973262306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600dc402d80 con 0x5600dc40c3e0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7ff003e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feff40994a0 tx=0x7feff40ce360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2968ecf9a998da65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7fefefe00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40a0e80 con 0x5600dc40c3e0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7feff40ce500 con 0x5600dc40c3e0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feff409add0 con 0x5600dc40c3e0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7feff409b990 con 0x5600dc40c3e0 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7ff01418e740 10 client.26854 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5600dc402d80 con 0x5600dc40c3e0 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.528+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feff409add0 con 0x5600dc40c3e0 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7feff4087080 con 0x5600dc40c3e0 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 10 client.26854 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 10 client.26854 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 20 client.26854 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc48c170 0x5600dc48e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 4 client.26854 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600dc375320 con 0x5600dc48c170 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.529+0000 7ff009e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc48c170 0x5600dc48e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:37.532 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.530+0000 7ff009e00640 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc48c170 0x5600dc48e590 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cfeb93ada6b6d82 server_cookie=9324f37673986 in_seq=0 out_seq=0 2024-04-25T19:00:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.530+0000 7fefefe00640 10 client.26854 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7fefefe00640 1 -- 172.21.15.63:0/1973262306 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600dc375320 con 0x5600dc48c170 2024-04-25T19:00:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7fefefe00640 10 client.26854 handle_command_reply: tid=0 2024-04-25T19:00:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 client.26854 shutdown 2024-04-25T19:00:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 msgr2=0x5600dc41b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feff40994a0 tx=0x7feff40ce360 comp rx=0 tx=0).stop 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 shutdown_connections 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5600dc48c170 0x5600dc48e590 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5600dc40cb90 0x5600dc41b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5600dc40c3e0 0x5600dc41b290 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 --2- 172.21.15.63:0/1973262306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5600dc4019b0 0x5600dc41ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 >> 172.21.15.63:0/1973262306 conn(0x5600dc3fdec0 msgr2=0x5600dc40b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.779+0000 7fefefe00640 0 client.26854 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.780+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 shutdown_connections 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.780+0000 7ff01418e740 1 -- 172.21.15.63:0/1973262306 wait complete. 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.780+0000 7ff01418e740 20 client.26854 trim_cache size 0 max 16384 2024-04-25T19:00:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.781+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 msgr2=0x7feff00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.781+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 0x7feff00a61d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x5600dc402f50 tx=0x7feffc0084c0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.781+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 msgr2=0x7ff00c19ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.781+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 0x7ff00c19ac40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feff400e9c0 tx=0x7feff400ee80 comp rx=0 tx=0).stop 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 shutdown_connections 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7feff00a3d10 0x7feff00a61d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff00c1092c0 0x7ff00c19ac40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff00c19efc0 0x7ff00c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 --2- 172.21.15.63:0/449610496 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ff00c1041f0 0x7ff00c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.782+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 >> 172.21.15.63:0/449610496 conn(0x7ff00c0ffa30 msgr2=0x7ff00c101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.783+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 shutdown_connections 2024-04-25T19:00:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:37.783+0000 7ff00b200640 1 -- 172.21.15.63:0/449610496 wait complete. 2024-04-25T19:00:38.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 21892 37.11 MB/sec warmup 36 sec latency 127.323 ms 2024-04-25T19:00:38.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:38 smithi063 ceph-mon[28006]: pgmap v443: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-25T19:00:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:38 smithi070 ceph-mon[32820]: pgmap v443: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-25T19:00:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:38 smithi114 ceph-mon[32988]: pgmap v443: 97 pgs: 97 active+clean; 285 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 12 MiB/s wr, 552 op/s 2024-04-25T19:00:39.074 INFO:tasks.workunit.client.0.smithi063.stdout: 10 22225 36.82 MB/sec warmup 37 sec latency 241.870 ms 2024-04-25T19:00:40.074 INFO:tasks.workunit.client.0.smithi063.stdout: 10 23179 37.56 MB/sec warmup 38 sec latency 46.807 ms 2024-04-25T19:00:40.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:40 smithi063 ceph-mon[28006]: pgmap v444: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 24 MiB/s wr, 928 op/s 2024-04-25T19:00:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:40 smithi070 ceph-mon[32820]: pgmap v444: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 24 MiB/s wr, 928 op/s 2024-04-25T19:00:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:40 smithi114 ceph-mon[32988]: pgmap v444: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 24 MiB/s wr, 928 op/s 2024-04-25T19:00:41.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 23780 37.54 MB/sec warmup 39 sec latency 45.690 ms 2024-04-25T19:00:41.601 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.3 2024-04-25T19:00:41.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:00:42.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 24307 37.16 MB/sec warmup 40 sec latency 126.476 ms 2024-04-25T19:00:42.155 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:00:42.568 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:42 smithi063 ceph-mon[28006]: pgmap v445: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:42.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:42 smithi070 ceph-mon[32820]: pgmap v445: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:42 smithi114 ceph-mon[32988]: pgmap v445: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:43.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 25205 37.05 MB/sec warmup 41 sec latency 56.976 ms 2024-04-25T19:00:44.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 26055 37.57 MB/sec warmup 42 sec latency 140.374 ms 2024-04-25T19:00:44.649 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:44 smithi063 ceph-mon[28006]: pgmap v446: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:44 smithi070 ceph-mon[32820]: pgmap v446: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:44 smithi114 ceph-mon[32988]: pgmap v446: 97 pgs: 97 active+clean; 409 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 22 MiB/s wr, 577 op/s 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 -- 172.21.15.63:0/3878280674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030074c70 msgr2=0x7f3030075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 --2- 172.21.15.63:0/3878280674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030074c70 0x7f3030075050 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f301c0098f0 tx=0x7f301c009c00 comp rx=0 tx=0).stop 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 -- 172.21.15.63:0/3878280674 shutdown_connections 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 --2- 172.21.15.63:0/3878280674 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030074c70 0x7f3030075050 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 --2- 172.21.15.63:0/3878280674 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030113c10 0x7f3030116000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 --2- 172.21.15.63:0/3878280674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3030075590 0x7f30301136d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.071+0000 7f3035400700 1 -- 172.21.15.63:0/3878280674 >> 172.21.15.63:0/3878280674 conn(0x7f3030079ba0 msgr2=0x7f3030079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.072+0000 7f3035400700 1 -- 172.21.15.63:0/3878280674 shutdown_connections 2024-04-25T19:00:45.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.072+0000 7f3035400700 1 -- 172.21.15.63:0/3878280674 wait complete. 2024-04-25T19:00:45.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.073+0000 7f3035400700 1 Processor -- start 2024-04-25T19:00:45.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.073+0000 7f3035400700 1 -- start start 2024-04-25T19:00:45.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 26771 37.65 MB/sec warmup 43 sec latency 54.633 ms 2024-04-25T19:00:45.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030075590 0x7f30301a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3030113c10 0x7f303019f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3030119240 con 0x7f3030113c10 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30301190c0 con 0x7f3030074c70 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f3035400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f30301193c0 con 0x7f3030075590 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f302e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f302e800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:51568/0 (socket says 172.21.15.63:51568) 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.074+0000 7f302e800700 1 -- 172.21.15.63:0/2519592652 learned_addr learned my addr 172.21.15.63:0/2519592652 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:45.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302de00700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030075590 0x7f30301a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 -- 172.21.15.63:0/2519592652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030075590 msgr2=0x7f30301a59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030075590 0x7f30301a59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 -- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3030113c10 msgr2=0x7f303019f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3030113c10 0x7f303019f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 -- 172.21.15.63:0/2519592652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303019fd80 con 0x7f3030074c70 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.075+0000 7f302e800700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f301c009230 tx=0x7f301c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.076+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c005220 con 0x7f3030074c70 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.076+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f301c00b560 con 0x7f3030074c70 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.076+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c003b50 con 0x7f3030074c70 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.076+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30301a0010 con 0x7f3030074c70 2024-04-25T19:00:45.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.076+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30301a03e0 con 0x7f3030074c70 2024-04-25T19:00:45.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.078+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3030069730 con 0x7f3030074c70 2024-04-25T19:00:45.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.079+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f301c003d10 con 0x7f3030074c70 2024-04-25T19:00:45.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.081+0000 7f3027400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 0x7f30140a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:45.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.081+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f301c0d18b0 con 0x7f3030074c70 2024-04-25T19:00:45.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.082+0000 7f302de00700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 0x7f30140a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:45.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.083+0000 7f302de00700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 0x7f30140a62f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3028009550 tx=0x7f30280091f0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:45.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.090+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f301c013030 con 0x7f3030074c70 2024-04-25T19:00:45.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.564+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30301ac4b0 con 0x7f3030074c70 2024-04-25T19:00:45.566 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:00:45.567 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:58:44.004081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T19:00:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.564+0000 7f3027400700 1 -- 172.21.15.63:0/2519592652 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11877 (secure 0 0 0) 0x7f301c09df40 con 0x7f3030074c70 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 msgr2=0x7f30140a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 0x7f30140a62f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3028009550 tx=0x7f30280091f0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 msgr2=0x7f30301a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f301c009230 tx=0x7f301c0094a0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 shutdown_connections 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f30140a3e30 0x7f30140a62f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3030113c10 0x7f303019f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3030074c70 0x7f30301a5460 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 --2- 172.21.15.63:0/2519592652 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3030075590 0x7f30301a59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:45.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.570+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 >> 172.21.15.63:0/2519592652 conn(0x7f3030079ba0 msgr2=0x7f303010f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:45.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.572+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 shutdown_connections 2024-04-25T19:00:45.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:45.572+0000 7f3035400700 1 -- 172.21.15.63:0/2519592652 wait complete. 2024-04-25T19:00:45.577 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 48 2024-04-25T19:00:46.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 27411 37.55 MB/sec warmup 44 sec latency 52.732 ms 2024-04-25T19:00:46.444 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 90.3 --await-for 42.0' 2024-04-25T19:00:46.444 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 90.3 --await-for 42.0 2024-04-25T19:00:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:46 smithi070 ceph-mon[32820]: pgmap v447: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 24 MiB/s wr, 820 op/s 2024-04-25T19:00:46.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:46 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2519592652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:00:46.810 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:46 smithi063 ceph-mon[28006]: pgmap v447: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 24 MiB/s wr, 820 op/s 2024-04-25T19:00:46.810 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:46 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2519592652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:00:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:46 smithi114 ceph-mon[32988]: pgmap v447: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 24 MiB/s wr, 820 op/s 2024-04-25T19:00:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:46 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2519592652' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:00:47.008 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:00:47.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 28201 37.53 MB/sec warmup 45 sec latency 39.844 ms 2024-04-25T19:00:48.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 29159 37.89 MB/sec warmup 46 sec latency 86.722 ms 2024-04-25T19:00:48.607 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:48 smithi063 ceph-mon[28006]: pgmap v448: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 618 op/s 2024-04-25T19:00:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:48 smithi070 ceph-mon[32820]: pgmap v448: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 618 op/s 2024-04-25T19:00:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:48 smithi114 ceph-mon[32988]: pgmap v448: 97 pgs: 97 active+clean; 411 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 618 op/s 2024-04-25T19:00:49.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 29917 38.14 MB/sec warmup 47 sec latency 149.139 ms 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.727+0000 7f1150000700 1 -- 172.21.15.63:0/2751602711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481059e0 msgr2=0x7f1148105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.727+0000 7f1150000700 1 --2- 172.21.15.63:0/2751602711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481059e0 0x7f1148105e60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1134009990 tx=0x7f1134009ca0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 -- 172.21.15.63:0/2751602711 shutdown_connections 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 --2- 172.21.15.63:0/2751602711 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 --2- 172.21.15.63:0/2751602711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481079f0 0x7f11481054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 --2- 172.21.15.63:0/2751602711 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481059e0 0x7f1148105e60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 -- 172.21.15.63:0/2751602711 >> 172.21.15.63:0/2751602711 conn(0x7f11480788e0 msgr2=0x7f1148078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 -- 172.21.15.63:0/2751602711 shutdown_connections 2024-04-25T19:00:49.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.728+0000 7f1150000700 1 -- 172.21.15.63:0/2751602711 wait complete. 2024-04-25T19:00:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.729+0000 7f1150000700 1 Processor -- start 2024-04-25T19:00:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.729+0000 7f1150000700 1 -- start start 2024-04-25T19:00:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481059e0 0x7f114819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 0x7f11481a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11481191f0 con 0x7f11481059e0 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1148119070 con 0x7f114806a740 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f1150000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1148119370 con 0x7f11481079f0 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f114e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 0x7f11481a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f114e000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 0x7f11481a3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48716/0 (socket says 172.21.15.63:48716) 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f114e000700 1 -- 172.21.15.63:0/1836334657 learned_addr learned my addr 172.21.15.63:0/1836334657 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f114d600700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.730+0000 7f114cc00700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481059e0 0x7f114819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 -- 172.21.15.63:0/1836334657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 msgr2=0x7f11481a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 0x7f11481a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 -- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481059e0 msgr2=0x7f114819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481059e0 0x7f114819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 -- 172.21.15.63:0/1836334657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11481a4690 con 0x7f114806a740 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f114d600700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114819f680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f113c00e9f0 tx=0x7f113c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f1146000700 1 -- 172.21.15.63:0/1836334657 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c00c9c0 con 0x7f114806a740 2024-04-25T19:00:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f1146000700 1 -- 172.21.15.63:0/1836334657 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f113c00cb60 con 0x7f114806a740 2024-04-25T19:00:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f1146000700 1 -- 172.21.15.63:0/1836334657 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c018680 con 0x7f114806a740 2024-04-25T19:00:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.731+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11481a4980 con 0x7f114806a740 2024-04-25T19:00:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.732+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11481ac200 con 0x7f114806a740 2024-04-25T19:00:49.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.736+0000 7f1146000700 1 -- 172.21.15.63:0/1836334657 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f113c010430 con 0x7f114806a740 2024-04-25T19:00:49.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.737+0000 7f1146000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 0x7f11300a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.738+0000 7f1146000700 1 -- 172.21.15.63:0/1836334657 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f113c0d0880 con 0x7f114806a740 2024-04-25T19:00:49.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.738+0000 7f114cc00700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 0x7f11300a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.738+0000 7f115a9cbb80 1 Processor -- start 2024-04-25T19:00:49.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.738+0000 7f115a9cbb80 1 -- start start 2024-04-25T19:00:49.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447cf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559544715020 con 0x5595447c1dd0 2024-04-25T19:00:49.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5595446bb8c0 con 0x5595447c26f0 2024-04-25T19:00:49.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559544710fe0 con 0x5595447cd560 2024-04-25T19:00:49.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f114cc00700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 0x7f11300a6390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11340098e0 tx=0x7f1134005960 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f114cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f114cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38684/0 (socket says 172.21.15.63:38684) 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.739+0000 7f114cc00700 1 -- 172.21.15.63:0/2815053785 learned_addr learned my addr 172.21.15.63:0/2815053785 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114e000700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114d600700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114cc00700 1 -- 172.21.15.63:0/2815053785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 msgr2=0x5595447cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114cc00700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114cc00700 1 -- 172.21.15.63:0/2815053785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 msgr2=0x5595447c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114cc00700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114cc00700 1 -- 172.21.15.63:0/2815053785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595446b1810 con 0x5595447c26f0 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.740+0000 7f114d600700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:49.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.741+0000 7f114cc00700 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1134015000 tx=0x7f113401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9684f9d590d8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.741+0000 7f113aa00700 1 -- 172.21.15.63:0/2815053785 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11340056d0 con 0x5595447c26f0 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.741+0000 7f113aa00700 1 -- 172.21.15.63:0/2815053785 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1134003680 con 0x5595447c26f0 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.741+0000 7f113aa00700 1 -- 172.21.15.63:0/2815053785 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134030790 con 0x5595447c26f0 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2815053785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 msgr2=0x5595447cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1134015000 tx=0x7f113401f000 comp rx=0 tx=0).stop 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2815053785 shutdown_connections 2024-04-25T19:00:49.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2815053785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447cd020 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.742+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2815053785 >> 172.21.15.63:0/2815053785 conn(0x5595446f10e0 msgr2=0x5595447be2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:49.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.743+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2815053785 shutdown_connections 2024-04-25T19:00:49.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.743+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2815053785 wait complete. 2024-04-25T19:00:49.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.743+0000 7f115a9cbb80 1 -- start start 2024-04-25T19:00:49.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 Processor -- start 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5595447cff30 con 0x5595447c1dd0 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5595446ce9f0 con 0x5595447c26f0 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5595447d00b0 con 0x5595447cd560 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.744+0000 7f115a9cbb80 10 client.0 _ll_register_callbacks cb 0x5595446ad8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114d600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34158/0 (socket says 172.21.15.63:34158) 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114d600700 1 -- 172.21.15.63:0/1883757542 learned_addr learned my addr 172.21.15.63:0/1883757542 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 -- 172.21.15.63:0/1883757542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 msgr2=0x5595447ca490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447ca490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 -- 172.21.15.63:0/1883757542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 msgr2=0x5595447ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.745+0000 7f114cc00700 1 -- 172.21.15.63:0/1883757542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559544662a60 con 0x5595447c26f0 2024-04-25T19:00:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f114d600700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:49.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f114cc00700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447c9f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1134009990 tx=0x7f1134003ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b022d4bfbcf23fb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f112f400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11340056d0 con 0x5595447c26f0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f1134030ca0 con 0x5595447c26f0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1134003680 con 0x5595447c26f0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f1134030460 con 0x5595447c26f0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f115a9cbb80 10 client.26855 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.746+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5595446abc80 con 0x5595447c26f0 2024-04-25T19:00:49.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.747+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1134030600 con 0x5595447c26f0 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.747+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f113400fc10 con 0x5595447c26f0 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.747+0000 7f115a9cbb80 10 client.26855 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.747+0000 7f115a9cbb80 10 client.26855 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.747+0000 7f115a9cbb80 20 client.26855 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.748+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5595447dcae0 0x5595447deed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.748+0000 7f115a9cbb80 4 client.26855 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.748+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1134003680 con 0x5595447dcae0 2024-04-25T19:00:49.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.748+0000 7f114d600700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5595447dcae0 0x5595447deed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.749+0000 7f114d600700 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5595447dcae0 0x5595447deed0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56ae084b1c8e2630 server_cookie=bc74824b985d5c78 in_seq=0 out_seq=0 2024-04-25T19:00:49.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.749+0000 7f112f400700 10 client.26855 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:49.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.751+0000 7f112f400700 1 -- 172.21.15.63:0/1883757542 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f1134003680 con 0x5595447dcae0 2024-04-25T19:00:49.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.751+0000 7f112f400700 10 client.26855 handle_command_reply: tid=0 2024-04-25T19:00:49.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.751+0000 7f115a9cbb80 1 client.26855 shutdown 2024-04-25T19:00:49.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 msgr2=0x5595447c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447c9f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1134009990 tx=0x7f1134003ae0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 shutdown_connections 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5595447dcae0 0x5595447deed0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c1dd0 0x5595447ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/1883757542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447c9f50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 >> 172.21.15.63:0/1883757542 conn(0x5595446f10e0 msgr2=0x5595447bded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f112f400700 0 client.26855 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 shutdown_connections 2024-04-25T19:00:49.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.752+0000 7f115a9cbb80 1 -- 172.21.15.63:0/1883757542 wait complete. 2024-04-25T19:00:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.753+0000 7f115a9cbb80 20 client.26855 trim_cache size 0 max 16384 2024-04-25T19:00:49.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 Processor -- start 2024-04-25T19:00:49.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 -- start start 2024-04-25T19:00:49.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 0x5595447be4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447cd560 0x5595447e2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b6360 con 0x5595447cd560 2024-04-25T19:00:49.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b3b30 con 0x5595447c26f0 2024-04-25T19:00:49.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b19a0 con 0x5595447c1dd0 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.786+0000 7f114cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114d600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 0x5595447be4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38694/0 (socket says 172.21.15.63:38694) 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 -- 172.21.15.63:0/4185495444 learned_addr learned my addr 172.21.15.63:0/4185495444 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114e000700 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447cd560 0x5595447e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 -- 172.21.15.63:0/4185495444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 msgr2=0x5595447be4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 0x5595447be4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 -- 172.21.15.63:0/4185495444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447cd560 msgr2=0x5595447e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447cd560 0x5595447e2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114cc00700 1 -- 172.21.15.63:0/4185495444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595447c84c0 con 0x5595447c26f0 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.787+0000 7f114d600700 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 0x5595447be4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:49.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.788+0000 7f114cc00700 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f113403a670 tx=0x7f1134005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898ac0f342da77de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.788+0000 7f113a000700 1 -- 172.21.15.63:0/4185495444 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134003680 con 0x5595447c26f0 2024-04-25T19:00:49.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.788+0000 7f113a000700 1 -- 172.21.15.63:0/4185495444 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1134044020 con 0x5595447c26f0 2024-04-25T19:00:49.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.788+0000 7f113a000700 1 -- 172.21.15.63:0/4185495444 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113403a7e0 con 0x5595447c26f0 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 -- 172.21.15.63:0/4185495444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 msgr2=0x5595447e1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f113403a670 tx=0x7f1134005fd0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 -- 172.21.15.63:0/4185495444 shutdown_connections 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447c1dd0 0x5595447be4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447cd560 0x5595447e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/4185495444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c26f0 0x5595447e1b20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 -- 172.21.15.63:0/4185495444 >> 172.21.15.63:0/4185495444 conn(0x5595446f10e0 msgr2=0x5595447d83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.789+0000 7f115a9cbb80 1 -- 172.21.15.63:0/4185495444 shutdown_connections 2024-04-25T19:00:49.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.790+0000 7f115a9cbb80 1 -- 172.21.15.63:0/4185495444 wait complete. 2024-04-25T19:00:49.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.791+0000 7f115a9cbb80 1 -- start start 2024-04-25T19:00:49.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.791+0000 7f115a9cbb80 1 Processor -- start 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c26f0 0x5595447e0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447e0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b1ff0 con 0x5595447c26f0 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b3460 con 0x5595447c1dd0 2024-04-25T19:00:49.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f11481b26c0 con 0x5595447cd560 2024-04-25T19:00:49.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f115a9cbb80 10 client.0 _ll_register_callbacks cb 0x5595446ad8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:49.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f114d600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f114d600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38706/0 (socket says 172.21.15.63:38706) 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f114d600700 1 -- 172.21.15.63:0/2572266892 learned_addr learned my addr 172.21.15.63:0/2572266892 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f114cc00700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c26f0 0x5595447e0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.792+0000 7f114e000700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447e0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 -- 172.21.15.63:0/2572266892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 msgr2=0x5595447e0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447e0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 -- 172.21.15.63:0/2572266892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c26f0 msgr2=0x5595447e0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c26f0 0x5595447e0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 -- 172.21.15.63:0/2572266892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559544842d00 con 0x5595447c1dd0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f114d600700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f113c00e9c0 tx=0x7f113c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d0aeb1b7b797514 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f112f400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c0870b0 con 0x5595447c1dd0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f113c09ca40 con 0x5595447c1dd0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1134044020 con 0x5595447c1dd0 2024-04-25T19:00:49.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f113c09cc80 con 0x5595447c1dd0 2024-04-25T19:00:49.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f115a9cbb80 10 client.26861 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:49.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.793+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559544844040 con 0x5595447c1dd0 2024-04-25T19:00:49.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1134044020 con 0x5595447c1dd0 2024-04-25T19:00:49.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f113c09db10 con 0x5595447c1dd0 2024-04-25T19:00:49.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 10 client.26861 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 10 client.26861 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 20 client.26861 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559544844770 0x559544846b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 4 client.26861 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.3, "await_for": 42.0}] 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 90.3, "await_for": 42.0}) -- 0x559544722810 con 0x559544844770 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.794+0000 7f114cc00700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559544844770 0x559544846b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.795+0000 7f114cc00700 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559544844770 0x559544846b90 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6153f5b018ffc0f4 server_cookie=7be6049804d8e345 in_seq=0 out_seq=0 2024-04-25T19:00:49.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:49.795+0000 7f112f400700 10 client.26861 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:50.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30362 37.72 MB/sec warmup 48 sec latency 260.285 ms 2024-04-25T19:00:50.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.379+0000 7f112f400700 1 -- 172.21.15.63:0/2572266892 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x559544722810 con 0x559544844770 2024-04-25T19:00:50.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.379+0000 7f112f400700 10 client.26861 handle_command_reply: tid=0 2024-04-25T19:00:50.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.379+0000 7f115a9cbb80 1 client.26861 shutdown 2024-04-25T19:00:50.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 msgr2=0x5595447e0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f113c00e9c0 tx=0x7f113c01a040 comp rx=0 tx=0).stop 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 shutdown_connections 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559544844770 0x559544846b90 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5595447cd560 0x5595447e0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5595447c1dd0 0x5595447e0520 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 --2- 172.21.15.63:0/2572266892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5595447c26f0 0x5595447e0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 >> 172.21.15.63:0/2572266892 conn(0x5595446f10e0 msgr2=0x5595447d8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.380+0000 7f112f400700 0 client.26861 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:50.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.381+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 shutdown_connections 2024-04-25T19:00:50.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.381+0000 7f115a9cbb80 1 -- 172.21.15.63:0/2572266892 wait complete. 2024-04-25T19:00:50.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.381+0000 7f115a9cbb80 20 client.26861 trim_cache size 0 max 16384 2024-04-25T19:00:50.383 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 48, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 42, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "a638040e": { 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 42, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 90.3, 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:00:50.384 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:00:50.385 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:00:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.385+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 msgr2=0x7f11300a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.385+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 0x7f11300a6390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11340098e0 tx=0x7f1134005960 comp rx=0 tx=0).stop 2024-04-25T19:00:50.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.385+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 msgr2=0x7f114819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.385+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114819f680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f113c00e9f0 tx=0x7f113c00ed00 comp rx=0 tx=0).stop 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 shutdown_connections 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f11300a3ed0 0x7f11300a6390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f11481079f0 0x7f11481a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f114806a740 0x7f114819f680 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 --2- 172.21.15.63:0/1836334657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f11481059e0 0x7f114819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.386+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 >> 172.21.15.63:0/1836334657 conn(0x7f11480788e0 msgr2=0x7f11480ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.387+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 shutdown_connections 2024-04-25T19:00:50.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:00:50.387+0000 7f1150000700 1 -- 172.21.15.63:0/1836334657 wait complete. 2024-04-25T19:00:50.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:50 smithi070 ceph-mon[32820]: pgmap v449: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 853 op/s 2024-04-25T19:00:50.769 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:50 smithi063 ceph-mon[28006]: pgmap v449: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 853 op/s 2024-04-25T19:00:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:50 smithi114 ceph-mon[32988]: pgmap v449: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 853 op/s 2024-04-25T19:00:50.934 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.932+0000 7ffa31200640 1 -- 172.21.15.63:0/2854358988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 msgr2=0x7ffa2c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.934 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.932+0000 7ffa31200640 1 --2- 172.21.15.63:0/2854358988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c101290 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009900 tx=0x7ffa1c02a3f0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.934 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 -- 172.21.15.63:0/2854358988 shutdown_connections 2024-04-25T19:00:50.934 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 --2- 172.21.15.63:0/2854358988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c101860 0x7ffa2c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 --2- 172.21.15.63:0/2854358988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 0x7ffa2c10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 --2- 172.21.15.63:0/2854358988 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c101290 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 -- 172.21.15.63:0/2854358988 >> 172.21.15.63:0/2854358988 conn(0x7ffa2c0fc770 msgr2=0x7ffa2c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 -- 172.21.15.63:0/2854358988 shutdown_connections 2024-04-25T19:00:50.935 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.933+0000 7ffa31200640 1 -- 172.21.15.63:0/2854358988 wait complete. 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 Processor -- start 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 -- start start 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 0x7ffa2c102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c1045e0 0x7ffa2c1033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c110f30 con 0x7ffa2c105f80 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c110db0 con 0x7ffa2c1045e0 2024-04-25T19:00:50.937 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa31200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1110b0 con 0x7ffa2c100eb0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 0x7ffa2c102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.934+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48788/0 (socket says 172.21.15.63:48788) 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 -- 172.21.15.63:0/450014804 learned_addr learned my addr 172.21.15.63:0/450014804 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2be00640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c1045e0 0x7ffa2c1033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 -- 172.21.15.63:0/450014804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c1045e0 msgr2=0x7ffa2c1033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c1045e0 0x7ffa2c1033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 -- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 msgr2=0x7ffa2c102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 0x7ffa2c102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 -- 172.21.15.63:0/450014804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c1ab2d0 con 0x7ffa2c100eb0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.935+0000 7ffa2b400640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005ab0 tx=0x7ffa1c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.936+0000 7ffa23e00640 1 -- 172.21.15.63:0/450014804 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c03f070 con 0x7ffa2c100eb0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.936+0000 7ffa23e00640 1 -- 172.21.15.63:0/450014804 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c031b60 con 0x7ffa2c100eb0 2024-04-25T19:00:50.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.936+0000 7ffa23e00640 1 -- 172.21.15.63:0/450014804 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c035030 con 0x7ffa2c100eb0 2024-04-25T19:00:50.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.936+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2c1ab560 con 0x7ffa2c100eb0 2024-04-25T19:00:50.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.936+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2c1aba00 con 0x7ffa2c100eb0 2024-04-25T19:00:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.939+0000 7ffa23e00640 1 -- 172.21.15.63:0/450014804 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7ffa1c042070 con 0x7ffa2c100eb0 2024-04-25T19:00:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.939+0000 7ffa357b0740 1 Processor -- start 2024-04-25T19:00:50.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.939+0000 7ffa357b0740 1 -- start start 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b8dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208b98650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559208af3900 con 0x559208b8d9b0 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559208a99ef0 con 0x559208b98b90 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa357b0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559208aef8c0 con 0x559208b8e2d0 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b8dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208b98650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208b98650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48798/0 (socket says 172.21.15.63:48798) 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2b400640 1 -- 172.21.15.63:0/1519775243 learned_addr learned my addr 172.21.15.63:0/1519775243 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2be00640 1 -- 172.21.15.63:0/1519775243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 msgr2=0x559208b98650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2be00640 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208b98650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.940+0000 7ffa2be00640 1 -- 172.21.15.63:0/1519775243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 msgr2=0x559208b9af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa2be00640 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa2be00640 1 -- 172.21.15.63:0/1519775243 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559208806210 con 0x559208b8d9b0 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa23e00640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 0x7ffa100a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa23e00640 1 -- 172.21.15.63:0/450014804 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7ffa1c0f0e40 con 0x7ffa2c100eb0 2024-04-25T19:00:50.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa2b400640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 0x7ffa100a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa2be00640 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b8dd90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffa24005fd0 tx=0x7ffa24011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f29a0baef49e153 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.941+0000 7ffa20c00640 1 -- 172.21.15.63:0/1519775243 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24004d20 con 0x559208b8d9b0 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa20c00640 1 -- 172.21.15.63:0/1519775243 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa24004050 con 0x559208b8d9b0 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa20c00640 1 -- 172.21.15.63:0/1519775243 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24005280 con 0x559208b8d9b0 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 -- 172.21.15.63:0/1519775243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 msgr2=0x559208b8dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa2b400640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 0x7ffa100a6250 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0b9250 tx=0x7ffa1c0f48f0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b8dd90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ffa24005fd0 tx=0x7ffa24011810 comp rx=0 tx=0).stop 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 -- 172.21.15.63:0/1519775243 shutdown_connections 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208b98650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 --2- 172.21.15.63:0/1519775243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b8dd90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.942+0000 7ffa357b0740 1 -- 172.21.15.63:0/1519775243 >> 172.21.15.63:0/1519775243 conn(0x559208b89ec0 msgr2=0x559208b8a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.943+0000 7ffa357b0740 1 -- 172.21.15.63:0/1519775243 shutdown_connections 2024-04-25T19:00:50.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.943+0000 7ffa357b0740 1 -- 172.21.15.63:0/1519775243 wait complete. 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.943+0000 7ffa357b0740 1 -- start start 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.943+0000 7ffa357b0740 1 Processor -- start 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b95b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b98b90 0x559208b965e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559208b9b7c0 con 0x559208b8d9b0 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x559208aad310 con 0x559208b8e2d0 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x559208b9b940 con 0x559208b98b90 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa357b0740 10 client.0 _ll_register_callbacks cb 0x559208a7bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b95b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38738/0 (socket says 172.21.15.63:38738) 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 -- 172.21.15.63:0/3473378684 learned_addr learned my addr 172.21.15.63:0/3473378684 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2be00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b98b90 0x559208b965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 -- 172.21.15.63:0/3473378684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b98b90 msgr2=0x559208b965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b98b90 0x559208b965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 -- 172.21.15.63:0/3473378684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 msgr2=0x559208b95b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b95b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2aa00640 1 -- 172.21.15.63:0/3473378684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592088d2620 con 0x559208b8e2d0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.944+0000 7ffa2be00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b95b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa180126d0 tx=0x7ffa18012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ddf0f8d2770fc3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa15600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18004030 con 0x559208b8e2d0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7ffa180041d0 con 0x559208b8e2d0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa24004050 con 0x559208b8e2d0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ffa180043d0 con 0x559208b8e2d0 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa357b0740 10 client.26873 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.945+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559208ba7ad0 con 0x559208b8e2d0 2024-04-25T19:00:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa24004050 con 0x559208b8e2d0 2024-04-25T19:00:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7ffa18003b60 con 0x559208b8e2d0 2024-04-25T19:00:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 10 client.26873 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 10 client.26873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 20 client.26873 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208ba8fe0 0x559208bab410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 4 client.26873 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559208b078b0 con 0x559208ba8fe0 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.946+0000 7ffa2be00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208ba8fe0 0x559208bab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.947+0000 7ffa2be00640 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208ba8fe0 0x559208bab410 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=222f0228efb1ff91 server_cookie=692182918981e6a4 in_seq=0 out_seq=0 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.947+0000 7ffa15600640 10 client.26873 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa15600640 1 -- 172.21.15.63:0/3473378684 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559208b078b0 con 0x559208ba8fe0 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa15600640 10 client.26873 handle_command_reply: tid=0 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 client.26873 shutdown 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 msgr2=0x559208b960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffa180126d0 tx=0x7ffa18012b90 comp rx=0 tx=0).stop 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 shutdown_connections 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208ba8fe0 0x559208bab410 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b8e2d0 0x559208b960a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b98b90 0x559208b965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 --2- 172.21.15.63:0/3473378684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208b95b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 >> 172.21.15.63:0/3473378684 conn(0x559208b89ec0 msgr2=0x559208b8fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa15600640 0 client.26873 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 shutdown_connections 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.948+0000 7ffa357b0740 1 -- 172.21.15.63:0/3473378684 wait complete. 2024-04-25T19:00:50.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.949+0000 7ffa357b0740 20 client.26873 trim_cache size 0 max 16384 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.965+0000 7ffa357b0740 1 Processor -- start 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.965+0000 7ffa357b0740 1 -- start start 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8d9b0 0x559208ba4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208bb1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b6ab0 con 0x559208b8e2d0 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b42c0 con 0x559208b98b90 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa357b0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b2130 con 0x559208b8d9b0 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208bb1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34226/0 (socket says 172.21.15.63:34226) 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2aa00640 1 -- 172.21.15.63:0/633723866 learned_addr learned my addr 172.21.15.63:0/633723866 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8d9b0 0x559208ba4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 -- 172.21.15.63:0/633723866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8d9b0 msgr2=0x559208ba4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8d9b0 0x559208ba4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 -- 172.21.15.63:0/633723866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 msgr2=0x559208bb0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2be00640 1 -- 172.21.15.63:0/633723866 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559208b899f0 con 0x559208b98b90 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.966+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.967+0000 7ffa2be00640 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208bb1220 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ffa2401b040 tx=0x7ffa24003f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=635ef449d075e7eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.967+0000 7ffa17e00640 1 -- 172.21.15.63:0/633723866 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24004590 con 0x559208b98b90 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.967+0000 7ffa17e00640 1 -- 172.21.15.63:0/633723866 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa24004050 con 0x559208b98b90 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.967+0000 7ffa17e00640 1 -- 172.21.15.63:0/633723866 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa240286b0 con 0x559208b98b90 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- 172.21.15.63:0/633723866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 msgr2=0x559208bb1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208bb1220 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ffa2401b040 tx=0x7ffa24003f50 comp rx=0 tx=0).stop 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- 172.21.15.63:0/633723866 shutdown_connections 2024-04-25T19:00:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208bb1220 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8e2d0 0x559208bb0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 --2- 172.21.15.63:0/633723866 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8d9b0 0x559208ba4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- 172.21.15.63:0/633723866 >> 172.21.15.63:0/633723866 conn(0x559208b89ec0 msgr2=0x559208b90e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- 172.21.15.63:0/633723866 shutdown_connections 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- 172.21.15.63:0/633723866 wait complete. 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 -- start start 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.968+0000 7ffa357b0740 1 Processor -- start 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208ba1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b2780 con 0x559208b8d9b0 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b3bf0 con 0x559208b98b90 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1b2e50 con 0x559208b8e2d0 2024-04-25T19:00:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa357b0740 10 client.0 _ll_register_callbacks cb 0x7ffa18009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208ba1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34238/0 (socket says 172.21.15.63:34238) 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.969+0000 7ffa2be00640 1 -- 172.21.15.63:0/2520255568 learned_addr learned my addr 172.21.15.63:0/2520255568 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2be00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 -- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 msgr2=0x559208b9e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 -- 172.21.15.63:0/2520255568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 msgr2=0x559208ba0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 -- 172.21.15.63:0/2520255568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559208b9e910 con 0x559208b8e2d0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2be00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa2aa00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208ba1090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ffa18010520 tx=0x7ffa180042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18df2418dff99564 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa15600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:00:50.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18031060 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa24004050 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7ffa18010a30 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.970+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7ffa18028040 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 10 client.26893 fetch_fsmap learned FSMap version 48 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x559208c0f180 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa24004050 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7ffa1802b030 con 0x559208b8e2d0 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 10 client.26893 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 10 client.26893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 20 client.26893 populate_metadata read hostname 'smithi063' 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208c0f940 0x559208c11d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 4 client.26893 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:00:50.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559208b01320 con 0x559208c0f940 2024-04-25T19:00:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.971+0000 7ffa2be00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208c0f940 0x559208c11d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:00:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.972+0000 7ffa2be00640 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208c0f940 0x559208c11d60 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=423eab2c37b64944 server_cookie=85f1d431778089b in_seq=0 out_seq=0 2024-04-25T19:00:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:50.972+0000 7ffa15600640 10 client.26893 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:51.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 36.95 MB/sec warmup 49 sec latency 1212.372 ms 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa15600640 1 -- 172.21.15.63:0/2520255568 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559208b01320 con 0x559208c0f940 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa15600640 10 client.26893 handle_command_reply: tid=0 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 client.26893 shutdown 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 msgr2=0x559208ba1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208ba1090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ffa18010520 tx=0x7ffa180042b0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 shutdown_connections 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x559208c0f940 0x559208c11d60 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x559208b98b90 0x559208b9e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x559208b8e2d0 0x559208ba1090 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 --2- 172.21.15.63:0/2520255568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559208b8d9b0 0x559208ba0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.187 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa15600640 0 client.26893 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 >> 172.21.15.63:0/2520255568 conn(0x559208b89ec0 msgr2=0x559208b88790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 shutdown_connections 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 1 -- 172.21.15.63:0/2520255568 wait complete. 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.185+0000 7ffa357b0740 20 client.26893 trim_cache size 0 max 16384 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:00:51.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 msgr2=0x7ffa100a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 0x7ffa100a6250 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c0b9250 tx=0x7ffa1c0f48f0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 msgr2=0x7ffa2c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c005ab0 tx=0x7ffa1c003c10 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 shutdown_connections 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7ffa100a3d90 0x7ffa100a6250 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa2c1045e0 0x7ffa2c1033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2c105f80 0x7ffa2c102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 --2- 172.21.15.63:0/450014804 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7ffa2c100eb0 0x7ffa2c102940 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:00:51.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 >> 172.21.15.63:0/450014804 conn(0x7ffa2c0fc770 msgr2=0x7ffa2c069ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:00:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 shutdown_connections 2024-04-25T19:00:51.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:00:51.187+0000 7ffa31200640 1 -- 172.21.15.63:0/450014804 wait complete. 2024-04-25T19:00:51.221 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'a638040e', quiesce took 9.6 seconds. Will release after: 30.3 2024-04-25T19:00:52.075 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 36.21 MB/sec warmup 50 sec latency 2212.461 ms 2024-04-25T19:00:52.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:52 smithi063 ceph-mon[28006]: pgmap v450: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 477 op/s 2024-04-25T19:00:52.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:52 smithi070 ceph-mon[32820]: pgmap v450: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 477 op/s 2024-04-25T19:00:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:52 smithi114 ceph-mon[32988]: pgmap v450: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 477 op/s 2024-04-25T19:00:53.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 35.50 MB/sec warmup 51 sec latency 3212.626 ms 2024-04-25T19:00:54.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 34.82 MB/sec warmup 52 sec latency 4212.732 ms 2024-04-25T19:00:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:54 smithi070 ceph-mon[32820]: pgmap v451: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-25T19:00:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:54 smithi114 ceph-mon[32988]: pgmap v451: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-25T19:00:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:54 smithi063 ceph-mon[28006]: pgmap v451: 97 pgs: 97 active+clean; 450 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-25T19:00:55.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 34.16 MB/sec warmup 53 sec latency 5212.871 ms 2024-04-25T19:00:56.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 33.53 MB/sec warmup 54 sec latency 6213.057 ms 2024-04-25T19:00:56.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:56 smithi070 ceph-mon[32820]: pgmap v452: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 706 op/s 2024-04-25T19:00:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:56 smithi114 ceph-mon[32988]: pgmap v452: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 706 op/s 2024-04-25T19:00:56.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:56 smithi063 ceph-mon[28006]: pgmap v452: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 706 op/s 2024-04-25T19:00:57.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 32.92 MB/sec warmup 55 sec latency 7213.210 ms 2024-04-25T19:00:58.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 32.33 MB/sec warmup 56 sec latency 8213.362 ms 2024-04-25T19:00:58.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:00:58 smithi070 ceph-mon[32820]: pgmap v453: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.1 MiB/s wr, 463 op/s 2024-04-25T19:00:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:00:58 smithi114 ceph-mon[32988]: pgmap v453: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.1 MiB/s wr, 463 op/s 2024-04-25T19:00:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:00:58 smithi063 ceph-mon[28006]: pgmap v453: 97 pgs: 97 active+clean; 433 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.1 MiB/s wr, 463 op/s 2024-04-25T19:00:59.076 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 31.77 MB/sec warmup 57 sec latency 9213.496 ms 2024-04-25T19:01:00.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 31.22 MB/sec warmup 58 sec latency 10213.634 ms 2024-04-25T19:01:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:00 smithi070 ceph-mon[32820]: pgmap v454: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 542 op/s 2024-04-25T19:01:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:00 smithi114 ceph-mon[32988]: pgmap v454: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 542 op/s 2024-04-25T19:01:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:00 smithi063 ceph-mon[28006]: pgmap v454: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 542 op/s 2024-04-25T19:01:01.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 30.69 MB/sec warmup 59 sec latency 11213.770 ms 2024-04-25T19:01:02.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 30.18 MB/sec warmup 60 sec latency 12213.939 ms 2024-04-25T19:01:02.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:02 smithi063 ceph-mon[28006]: pgmap v455: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:02 smithi070 ceph-mon[32820]: pgmap v455: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:02 smithi114 ceph-mon[32988]: pgmap v455: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:03.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 29.68 MB/sec warmup 61 sec latency 13214.088 ms 2024-04-25T19:01:04.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 29.20 MB/sec warmup 62 sec latency 14214.188 ms 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 -- 172.21.15.63:0/481603663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 msgr2=0x7f61a810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 --2- 172.21.15.63:0/481603663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a810e050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6190009900 tx=0x7f619002a480 comp rx=0 tx=0).stop 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 -- 172.21.15.63:0/481603663 shutdown_connections 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 --2- 172.21.15.63:0/481603663 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a810e590 0x7f61a81109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 --2- 172.21.15.63:0/481603663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a810e050 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 --2- 172.21.15.63:0/481603663 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a8074380 0x7f61a8074760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 -- 172.21.15.63:0/481603663 >> 172.21.15.63:0/481603663 conn(0x7f61a80fd8a0 msgr2=0x7f61a80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 -- 172.21.15.63:0/481603663 shutdown_connections 2024-04-25T19:01:04.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.335+0000 7f61adc00640 1 -- 172.21.15.63:0/481603663 wait complete. 2024-04-25T19:01:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.336+0000 7f61adc00640 1 Processor -- start 2024-04-25T19:01:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.336+0000 7f61adc00640 1 -- start start 2024-04-25T19:01:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.336+0000 7f61adc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 0x7f61a81a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.336+0000 7f61adc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61adc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a810e590 0x7f61a819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61adc00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61a8114220 con 0x7f61a8074ca0 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61adc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81140a0 con 0x7f61a8074380 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61adc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81143a0 con 0x7f61a810e590 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51320/0 (socket says 172.21.15.63:51320) 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7400640 1 -- 172.21.15.63:0/2423551294 learned_addr learned my addr 172.21.15.63:0/2423551294 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:04.342 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 0x7f61a81a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61ad200640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a810e590 0x7f61a819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 -- 172.21.15.63:0/2423551294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a810e590 msgr2=0x7f61a819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a810e590 0x7f61a819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 -- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 msgr2=0x7f61a81a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.343 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7e00640 1 -- 172.21.15.63:0/2423551294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a819ad20 con 0x7f61a8074380 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.337+0000 7f61a7400640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 0x7f61a81a03f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6198009fd0 tx=0x7f619800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61a4c00640 1 -- 172.21.15.63:0/2423551294 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198004b20 con 0x7f61a8074380 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a819b010 con 0x7f61a8074380 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61a4c00640 1 -- 172.21.15.63:0/2423551294 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6198004cc0 con 0x7f61a8074380 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61a4c00640 1 -- 172.21.15.63:0/2423551294 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198010720 con 0x7f61a8074380 2024-04-25T19:01:04.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.338+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a81a72b0 con 0x7f61a8074380 2024-04-25T19:01:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.341+0000 7f61b21db740 1 Processor -- start 2024-04-25T19:01:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 -- start start 2024-04-25T19:01:04.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa8593d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61a4c00640 1 -- 172.21.15.63:0/2423551294 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6198005020 con 0x7f61a8074380 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 0x558fa859e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558fa84f9900 con 0x558fa85942d0 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558fa849fef0 con 0x558fa859eb90 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61b21db740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558fa84f58c0 con 0x558fa85939b0 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61a7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 0x558fa859e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61a7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 0x558fa859e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51322/0 (socket says 172.21.15.63:51322) 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61a7e00640 1 -- 172.21.15.63:0/71729402 learned_addr learned my addr 172.21.15.63:0/71729402 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61ad200640 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa8593d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.342+0000 7f61a7400640 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 -- 172.21.15.63:0/71729402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 msgr2=0x558fa85a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 -- 172.21.15.63:0/71729402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 msgr2=0x558fa859e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 0x558fa859e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 -- 172.21.15.63:0/71729402 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa820c210 con 0x558fa85939b0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f61ad200640 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa8593d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009b80 tx=0x7f61a0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac502bb0d5ae55a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f619d600640 1 -- 172.21.15.63:0/71729402 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0004050 con 0x558fa85939b0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f619d600640 1 -- 172.21.15.63:0/71729402 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61a00054d0 con 0x558fa85939b0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.343+0000 7f619d600640 1 -- 172.21.15.63:0/71729402 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0004c40 con 0x558fa85939b0 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61a4c00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 0x7f618c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61a4c00640 1 -- 172.21.15.63:0/2423551294 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f61980d0fe0 con 0x7f61a8074380 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 0x7f618c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 -- 172.21.15.63:0/71729402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 msgr2=0x558fa8593d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa8593d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009b80 tx=0x7f61a0011810 comp rx=0 tx=0).stop 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 -- 172.21.15.63:0/71729402 shutdown_connections 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85942d0 0x558fa859e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 --2- 172.21.15.63:0/71729402 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa8593d90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.344+0000 7f61b21db740 1 -- 172.21.15.63:0/71729402 >> 172.21.15.63:0/71729402 conn(0x558fa858fec0 msgr2=0x558fa85902a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.345+0000 7f61b21db740 1 -- 172.21.15.63:0/71729402 shutdown_connections 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.345+0000 7f61b21db740 1 -- 172.21.15.63:0/71729402 wait complete. 2024-04-25T19:01:04.350 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.345+0000 7f61a7e00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 0x7f618c0a61d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f619809cc00 tx=0x7f61980d4ab0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.345+0000 7f61b21db740 1 -- start start 2024-04-25T19:01:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.345+0000 7f61b21db740 1 Processor -- start 2024-04-25T19:01:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa859bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa859c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558fa85a17c0 con 0x558fa859eb90 2024-04-25T19:01:04.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558fa84b3310 con 0x558fa85942d0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558fa85a1940 con 0x558fa85939b0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61b21db740 10 client.0 _ll_register_callbacks cb 0x558fa8481cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61ad200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa859bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51336/0 (socket says 172.21.15.63:51336) 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61ad200640 1 -- 172.21.15.63:0/1180819206 learned_addr learned my addr 172.21.15.63:0/1180819206 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.346+0000 7f61a7400640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa859c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 -- 172.21.15.63:0/1180819206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 msgr2=0x558fa859bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa859bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 -- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 msgr2=0x558fa859c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa859c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 -- 172.21.15.63:0/1180819206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa82d8620 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61ad200640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009b50 tx=0x7f61a0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70069a5942e62587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f6196000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a002d060 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f61a0004d80 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.347+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61a00054d0 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f61a0004f80 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 10 client.16692 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558fa85ada20 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61a00054d0 con 0x558fa859eb90 2024-04-25T19:01:04.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f61a0028450 con 0x558fa859eb90 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 10 client.16692 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 20 client.16692 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa85aee80 0x558fa85b1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 4 client.16692 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.348+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fa850d8b0 con 0x558fa85aee80 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.349+0000 7f61a7400640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa85aee80 0x558fa85b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.349+0000 7f61a7400640 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa85aee80 0x558fa85b1270 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f02fc7dde12ce11 server_cookie=81dd016fa1a0c97b in_seq=0 out_seq=0 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.349+0000 7f6196000640 10 client.16692 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:04.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f6196000640 1 -- 172.21.15.63:0/1180819206 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fa850d8b0 con 0x558fa85aee80 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f6196000640 10 client.16692 handle_command_reply: tid=0 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 client.16692 shutdown 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 msgr2=0x558fa859c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009b50 tx=0x7f61a0000bc0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 shutdown_connections 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa85aee80 0x558fa85b1270 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa859c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa859eb90 0x558fa859c5a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 --2- 172.21.15.63:0/1180819206 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85939b0 0x558fa859bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.359 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f6196000640 0 client.16692 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 >> 172.21.15.63:0/1180819206 conn(0x558fa858fec0 msgr2=0x558fa8595c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.350+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 shutdown_connections 2024-04-25T19:01:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.351+0000 7f61b21db740 1 -- 172.21.15.63:0/1180819206 wait complete. 2024-04-25T19:01:04.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.351+0000 7f61b21db740 20 client.16692 trim_cache size 0 max 16384 2024-04-25T19:01:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.367+0000 7f61b21db740 1 Processor -- start 2024-04-25T19:01:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.367+0000 7f61b21db740 1 -- start start 2024-04-25T19:01:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85aa680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa8596370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa859eb90 0x558fa859da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81b26f0 con 0x558fa85939b0 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81aff00 con 0x558fa85942d0 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61b21db740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81add70 con 0x558fa859eb90 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85aa680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51340/0 (socket says 172.21.15.63:51340) 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61ad200640 1 -- 172.21.15.63:0/177589472 learned_addr learned my addr 172.21.15.63:0/177589472 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:04.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa8596370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61ad200640 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa859eb90 0x558fa859da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 -- 172.21.15.63:0/177589472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa859eb90 msgr2=0x558fa859da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa859eb90 0x558fa859da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 -- 172.21.15.63:0/177589472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 msgr2=0x558fa85aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.368+0000 7f61a7400640 1 -- 172.21.15.63:0/177589472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fa858f970 con 0x558fa85942d0 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.369+0000 7f61a7400640 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa8596370 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6190000c00 tx=0x7f6190003940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a7266c02faa291 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.369+0000 7f619cc00640 1 -- 172.21.15.63:0/177589472 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190035040 con 0x558fa85942d0 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.369+0000 7f619cc00640 1 -- 172.21.15.63:0/177589472 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61900468b0 con 0x558fa85942d0 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.369+0000 7f619cc00640 1 -- 172.21.15.63:0/177589472 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190031aa0 con 0x558fa85942d0 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- 172.21.15.63:0/177589472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 msgr2=0x558fa8596370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa8596370 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6190000c00 tx=0x7f6190003940 comp rx=0 tx=0).stop 2024-04-25T19:01:04.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- 172.21.15.63:0/177589472 shutdown_connections 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa85942d0 0x558fa8596370 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa859eb90 0x558fa859da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 --2- 172.21.15.63:0/177589472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- 172.21.15.63:0/177589472 >> 172.21.15.63:0/177589472 conn(0x558fa858fec0 msgr2=0x558fa859d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- 172.21.15.63:0/177589472 shutdown_connections 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- 172.21.15.63:0/177589472 wait complete. 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 -- start start 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.370+0000 7f61b21db740 1 Processor -- start 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 0x558fa85a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81ae3c0 con 0x558fa85939b0 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81af830 con 0x558fa859eb90 2024-04-25T19:01:04.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81aea90 con 0x558fa85942d0 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61b21db740 10 client.0 _ll_register_callbacks cb 0x7f61a00098b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61ad200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51342/0 (socket says 172.21.15.63:51342) 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61ad200640 1 -- 172.21.15.63:0/1354055703 learned_addr learned my addr 172.21.15.63:0/1354055703 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61a7400640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 0x558fa85a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.371+0000 7f61ad200640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 -- 172.21.15.63:0/1354055703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 msgr2=0x558fa85a4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 0x558fa85a4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 -- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 msgr2=0x558fa85a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 -- 172.21.15.63:0/1354055703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fa86141e0 con 0x558fa85939b0 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61a7400640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 0x558fa85a4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61ad200640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61a0001eb0 tx=0x7f61a0009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e90740d4124d568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f6196000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a002d040 con 0x558fa85939b0 2024-04-25T19:01:04.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 8587+0+0 (secure 0 0 0) 0x7f61a0004460 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.372+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61900468b0 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f61a0004660 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 10 client.16698 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558fa8615380 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61900468b0 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f61a0003950 con 0x558fa85939b0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 10 client.16698 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 10 client.16698 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 20 client.16698 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa86159a0 0x558fa8617dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 4 client.16698 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.373+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558fa8507320 con 0x558fa86159a0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.374+0000 7f61a7400640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa86159a0 0x558fa8617dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.374+0000 7f61a7400640 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa86159a0 0x558fa8617dc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=263610d3b63dc221 server_cookie=1ca498e681d28529 in_seq=0 out_seq=0 2024-04-25T19:01:04.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.374+0000 7f6196000640 10 client.16698 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f6196000640 1 -- 172.21.15.63:0/1354055703 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558fa8507320 con 0x558fa86159a0 2024-04-25T19:01:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f6196000640 10 client.16698 handle_command_reply: tid=0 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 client.16698 shutdown 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 msgr2=0x558fa85a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61a0001eb0 tx=0x7f61a0009d30 comp rx=0 tx=0).stop 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 shutdown_connections 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558fa86159a0 0x558fa8617dc0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558fa859eb90 0x558fa85a4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558fa85942d0 0x558fa85a4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 --2- 172.21.15.63:0/1354055703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558fa85939b0 0x558fa85a4500 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.423+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 >> 172.21.15.63:0/1354055703 conn(0x558fa858fec0 msgr2=0x558fa859dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.424+0000 7f6196000640 0 client.16698 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.424+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 shutdown_connections 2024-04-25T19:01:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.424+0000 7f61b21db740 1 -- 172.21.15.63:0/1354055703 wait complete. 2024-04-25T19:01:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.424+0000 7f61b21db740 20 client.16698 trim_cache size 0 max 16384 2024-04-25T19:01:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:01:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:01:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:01:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 msgr2=0x7f618c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 0x7f618c0a61d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f619809cc00 tx=0x7f61980d4ab0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 msgr2=0x7f61a81a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 0x7f61a81a03f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6198009fd0 tx=0x7f619800ef90 comp rx=0 tx=0).stop 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 shutdown_connections 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f618c0a3d10 0x7f618c0a61d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f61a810e590 0x7f61a819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61a8074ca0 0x7f61a81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 --2- 172.21.15.63:0/2423551294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61a8074380 0x7f61a81a03f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.425+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 >> 172.21.15.63:0/2423551294 conn(0x7f61a80fd8a0 msgr2=0x7f61a80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.427+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 shutdown_connections 2024-04-25T19:01:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:04.427+0000 7f61adc00640 1 -- 172.21.15.63:0/2423551294 wait complete. 2024-04-25T19:01:04.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:04 smithi070 ceph-mon[32820]: pgmap v456: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:04 smithi114 ceph-mon[32988]: pgmap v456: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:04 smithi063 ceph-mon[28006]: pgmap v456: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 4.1 MiB/s wr, 307 op/s 2024-04-25T19:01:05.077 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 28.74 MB/sec warmup 63 sec latency 15214.325 ms 2024-04-25T19:01:06.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 28.29 MB/sec warmup 64 sec latency 16214.485 ms 2024-04-25T19:01:06.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:06 smithi070 ceph-mon[32820]: pgmap v457: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-25T19:01:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:06 smithi114 ceph-mon[32988]: pgmap v457: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-25T19:01:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:06 smithi063 ceph-mon[28006]: pgmap v457: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 7.4 MiB/s wr, 457 op/s 2024-04-25T19:01:07.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 27.86 MB/sec warmup 65 sec latency 17214.584 ms 2024-04-25T19:01:08.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 27.43 MB/sec warmup 66 sec latency 18214.680 ms 2024-04-25T19:01:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:08 smithi070 ceph-mon[32820]: pgmap v458: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T19:01:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:08 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:08 smithi114 ceph-mon[32988]: pgmap v458: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T19:01:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:08 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:08 smithi063 ceph-mon[28006]: pgmap v458: 97 pgs: 97 active+clean; 422 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 5.5 MiB/s wr, 228 op/s 2024-04-25T19:01:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:08 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:01:09.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 27.02 MB/sec warmup 67 sec latency 19214.799 ms 2024-04-25T19:01:09.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:09 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:09.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:09 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:09.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:09 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:09.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:09 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:09 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:01:09.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:09 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:01:10.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 26.63 MB/sec warmup 68 sec latency 20214.871 ms 2024-04-25T19:01:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:10 smithi070 ceph-mon[32820]: pgmap v459: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 9.3 MiB/s wr, 364 op/s 2024-04-25T19:01:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:10 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:01:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:10 smithi070 ceph-mon[32820]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T19:01:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:10 smithi114 ceph-mon[32988]: pgmap v459: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 9.3 MiB/s wr, 364 op/s 2024-04-25T19:01:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:10 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:01:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:10 smithi114 ceph-mon[32988]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T19:01:10.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:10 smithi063 ceph-mon[28006]: pgmap v459: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 9.3 MiB/s wr, 364 op/s 2024-04-25T19:01:10.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:10 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:01:10.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:10 smithi063 ceph-mon[28006]: osdmap e116: 12 total, 12 up, 12 in 2024-04-25T19:01:11.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 26.24 MB/sec warmup 69 sec latency 21214.945 ms 2024-04-25T19:01:12.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 25.87 MB/sec warmup 70 sec latency 22215.077 ms 2024-04-25T19:01:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:12 smithi070 ceph-mon[32820]: pgmap v461: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:12 smithi114 ceph-mon[32988]: pgmap v461: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:12 smithi063 ceph-mon[28006]: pgmap v461: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:13.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 25.50 MB/sec warmup 71 sec latency 23215.242 ms 2024-04-25T19:01:14.078 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 25.15 MB/sec warmup 72 sec latency 24215.369 ms 2024-04-25T19:01:14.815 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:14 smithi114 ceph-mon[32988]: pgmap v462: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:14 smithi063 ceph-mon[28006]: pgmap v462: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:14 smithi070 ceph-mon[32820]: pgmap v462: 97 pgs: 97 active+clean; 416 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 8.6 MiB/s wr, 342 op/s 2024-04-25T19:01:15.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 24.80 MB/sec warmup 73 sec latency 25215.550 ms 2024-04-25T19:01:16.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 24.47 MB/sec warmup 74 sec latency 26215.702 ms 2024-04-25T19:01:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:16 smithi114 ceph-mon[32988]: pgmap v463: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:16 smithi063 ceph-mon[28006]: pgmap v463: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:16 smithi070 ceph-mon[32820]: pgmap v463: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:17.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 24.14 MB/sec warmup 75 sec latency 27215.867 ms 2024-04-25T19:01:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.566+0000 7f9ebae00640 1 -- 172.21.15.63:0/2075945238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4104d70 msgr2=0x7f9eb4108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.566+0000 7f9ebae00640 1 --2- 172.21.15.63:0/2075945238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4104d70 0x7f9eb4108db0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009be0 tx=0x7f9ea002a6d0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.569 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 -- 172.21.15.63:0/2075945238 shutdown_connections 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 --2- 172.21.15.63:0/2075945238 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4104d70 0x7f9eb4108db0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 --2- 172.21.15.63:0/2075945238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb4109490 0x7f9eb4111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 --2- 172.21.15.63:0/2075945238 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb41043c0 0x7f9eb41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 -- 172.21.15.63:0/2075945238 >> 172.21.15.63:0/2075945238 conn(0x7f9eb40ffc20 msgr2=0x7f9eb4102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 -- 172.21.15.63:0/2075945238 shutdown_connections 2024-04-25T19:01:17.570 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.567+0000 7f9ebae00640 1 -- 172.21.15.63:0/2075945238 wait complete. 2024-04-25T19:01:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.568+0000 7f9ebae00640 1 Processor -- start 2024-04-25T19:01:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.568+0000 7f9ebae00640 1 -- start start 2024-04-25T19:01:17.571 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.568+0000 7f9ebae00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb41043c0 0x7f9eb41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9ebae00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9ebae00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 0x7f9eb419ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9ebae00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41143d0 con 0x7f9eb41043c0 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9ebae00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4114250 con 0x7f9eb4109490 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9ebae00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4114550 con 0x7f9eb419a790 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb41043c0 0x7f9eb41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41692/0 (socket says 172.21.15.63:41692) 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 -- 172.21.15.63:0/228871066 learned_addr learned my addr 172.21.15.63:0/228871066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 0x7f9eb419ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 -- 172.21.15.63:0/228871066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 msgr2=0x7f9eb419ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 0x7f9eb419ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 -- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb41043c0 msgr2=0x7f9eb41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb41043c0 0x7f9eb41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.572 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb3400640 1 -- 172.21.15.63:0/228871066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb419b320 con 0x7f9eb4109490 2024-04-25T19:01:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.569+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 0x7f9eb419ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9eb3400640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000f80 tx=0x7f9ea0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.573 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9eb0c00640 1 -- 172.21.15.63:0/228871066 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0041020 con 0x7f9eb4109490 2024-04-25T19:01:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9eb0c00640 1 -- 172.21.15.63:0/228871066 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea00318f0 con 0x7f9eb4109490 2024-04-25T19:01:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9eb0c00640 1 -- 172.21.15.63:0/228871066 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0036040 con 0x7f9eb4109490 2024-04-25T19:01:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb419b5b0 con 0x7f9eb4109490 2024-04-25T19:01:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.570+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb41a7710 con 0x7f9eb4109490 2024-04-25T19:01:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.573+0000 7f9ebdbaf740 1 Processor -- start 2024-04-25T19:01:17.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 -- start start 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb0c00640 1 -- 172.21.15.63:0/228871066 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9ea0044020 con 0x7f9eb4109490 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 0x561500f36c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f43df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561500e9c900 con 0x561500f37140 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561500e42ef0 con 0x561500f41a00 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561500e988c0 con 0x561500f36820 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 0x561500f36c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34736/0 (socket says 172.21.15.63:34736) 2024-04-25T19:01:17.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb3e00640 1 -- 172.21.15.63:0/3884271341 learned_addr learned my addr 172.21.15.63:0/3884271341 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.574+0000 7f9eb3400640 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f43df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 -- 172.21.15.63:0/3884271341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 msgr2=0x561500f43df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f43df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 -- 172.21.15.63:0/3884271341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 msgr2=0x561500f414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 -- 172.21.15.63:0/3884271341 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561500baf210 con 0x561500f36820 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 0x561500f36c00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009b90 tx=0x7f9eac0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75807b12b6200d44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9ea9600640 1 -- 172.21.15.63:0/3884271341 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac005020 con 0x561500f36820 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9ea9600640 1 -- 172.21.15.63:0/3884271341 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eac0047a0 con 0x561500f36820 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.575+0000 7f9ea9600640 1 -- 172.21.15.63:0/3884271341 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac011680 con 0x561500f36820 2024-04-25T19:01:17.579 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9eb0c00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 0x7f9e980a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9eb0c00640 1 -- 172.21.15.63:0/228871066 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8631+0+0 (secure 0 0 0) 0x7f9ea00f10e0 con 0x7f9eb4109490 2024-04-25T19:01:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3884271341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 msgr2=0x561500f36c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 0x561500f36c00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009b90 tx=0x7f9eac0107a0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.583 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 0x7f9e980a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3884271341 shutdown_connections 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f43df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f37140 0x561500f414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3884271341 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f36820 0x561500f36c00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3884271341 >> 172.21.15.63:0/3884271341 conn(0x561500f32d30 msgr2=0x561500f33110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3884271341 shutdown_connections 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.576+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3884271341 wait complete. 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.577+0000 7f9ebdbaf740 1 -- start start 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.577+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 0x7f9e980a6300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x561500f37dc0 tx=0x7f9ea4005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.577+0000 7f9ebdbaf740 1 Processor -- start 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.577+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f3ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f3f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561500f44630 con 0x561500f36820 2024-04-25T19:01:17.584 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561500e56310 con 0x561500f37140 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x561500f447b0 con 0x561500f41a00 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9ebdbaf740 10 client.0 _ll_register_callbacks cb 0x561500e24cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f3ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34748/0 (socket says 172.21.15.63:34748) 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 -- 172.21.15.63:0/3812124810 learned_addr learned my addr 172.21.15.63:0/3812124810 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f3f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 -- 172.21.15.63:0/3812124810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 msgr2=0x561500f3f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f3f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 -- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 msgr2=0x561500f3ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f3ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3e00640 1 -- 172.21.15.63:0/3812124810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561500c7b620 con 0x561500f36820 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.578+0000 7f9eb3400640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f3ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ea401e9d0 tx=0x7f9ea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0215ab1db10c3d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9e9e000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea402e060 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8631+0+0 (secure 0 0 0) 0x7f9ea4018030 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac0047a0 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4031420 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9ebdbaf740 10 client.16710 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.579+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x561500f50940 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eac0047a0 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f9ea401c640 con 0x561500f36820 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 10 client.16710 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 10 client.16710 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 20 client.16710 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500f51e50 0x561500f542c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 4 client.16710 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:01:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.580+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561500eb08b0 con 0x561500f51e50 2024-04-25T19:01:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.581+0000 7f9eb3400640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500f51e50 0x561500f542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.581+0000 7f9eb3400640 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500f51e50 0x561500f542c0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5deb480e8c1ee37 server_cookie=21065c7bcd2a935f in_seq=0 out_seq=0 2024-04-25T19:01:17.587 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.581+0000 7f9e9e000640 10 client.16710 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9e9e000640 1 -- 172.21.15.63:0/3812124810 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561500eb08b0 con 0x561500f51e50 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9e9e000640 10 client.16710 handle_command_reply: tid=0 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 client.16710 shutdown 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 msgr2=0x561500f3e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9ea401e9d0 tx=0x7f9ea40046c0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 shutdown_connections 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500f51e50 0x561500f542c0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f3ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.588 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f3f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/3812124810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f3e9d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9e9e000640 0 client.16710 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 >> 172.21.15.63:0/3812124810 conn(0x561500f32d30 msgr2=0x561500f38ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 shutdown_connections 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/3812124810 wait complete. 2024-04-25T19:01:17.589 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.582+0000 7f9ebdbaf740 20 client.16710 trim_cache size 0 max 16384 2024-04-25T19:01:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 Processor -- start 2024-04-25T19:01:17.602 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 -- start start 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 0x561500f3f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b28c0 con 0x561500f36820 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41b00d0 con 0x561500f41a00 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.599+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41adf40 con 0x561500f37140 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41728/0 (socket says 172.21.15.63:41728) 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 -- 172.21.15.63:0/39712499 learned_addr learned my addr 172.21.15.63:0/39712499 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb3400640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 0x561500f3f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 -- 172.21.15.63:0/39712499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 msgr2=0x561500f3f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 0x561500f3f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 -- 172.21.15.63:0/39712499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 msgr2=0x561500f391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.603 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb8e00640 1 -- 172.21.15.63:0/39712499 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561500f2e690 con 0x561500f41a00 2024-04-25T19:01:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.600+0000 7f9eb3400640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 0x561500f3f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:17.604 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9eac010ca0 tx=0x7f9eac0097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abc6c6065ecd4ec5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9ea8c00640 1 -- 172.21.15.63:0/39712499 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac02c070 con 0x561500f41a00 2024-04-25T19:01:17.607 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9ea8c00640 1 -- 172.21.15.63:0/39712499 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eac004b10 con 0x561500f41a00 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9ea8c00640 1 -- 172.21.15.63:0/39712499 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac0056b0 con 0x561500f41a00 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/39712499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 msgr2=0x561500f59bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.601+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9eac010ca0 tx=0x7f9eac0097f0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/39712499 shutdown_connections 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f41a00 0x561500f59bd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f37140 0x561500f3f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/39712499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/39712499 >> 172.21.15.63:0/39712499 conn(0x561500f32d30 msgr2=0x561500f3e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/39712499 shutdown_connections 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/39712499 wait complete. 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 -- start start 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.602+0000 7f9ebdbaf740 1 Processor -- start 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f47230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f47840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41ae590 con 0x561500f36820 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41afa00 con 0x561500f37140 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41aec60 con 0x561500f41a00 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9ebdbaf740 10 client.0 _ll_register_callbacks cb 0x7f9ea401e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9eb3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f47230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34768/0 (socket says 172.21.15.63:34768) 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9eb3e00640 1 -- 172.21.15.63:0/98809407 learned_addr learned my addr 172.21.15.63:0/98809407 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.603+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f47840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 -- 172.21.15.63:0/98809407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 msgr2=0x561500f47840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f47840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 -- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 msgr2=0x561500f47230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:17.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f47230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 -- 172.21.15.63:0/98809407 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561500f48010 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb8e00640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f47840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3400640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f47230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9eb3e00640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40096b0 tx=0x7f9ea401c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ac01751278aa52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9e9e000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea403f060 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 8631+0+0 (secure 0 0 0) 0x7f9ea4005250 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.604+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac004b10 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9ea402e020 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 10 client.16716 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x561500fb8530 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9eac004b10 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f9ea40315c0 con 0x561500f36820 2024-04-25T19:01:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 10 client.16716 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 10 client.16716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 20 client.16716 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500fb8d10 0x561500fbb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 4 client.16716 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.605+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561500eaa320 con 0x561500fb8d10 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.606+0000 7f9eb3400640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500fb8d10 0x561500fbb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.606+0000 7f9eb3400640 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500fb8d10 0x561500fbb130 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df3398e1ec294cc4 server_cookie=aa7bb491952b629f in_seq=0 out_seq=0 2024-04-25T19:01:17.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.606+0000 7f9e9e000640 10 client.16716 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9e9e000640 1 -- 172.21.15.63:0/98809407 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561500eaa320 con 0x561500fb8d10 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9e9e000640 10 client.16716 handle_command_reply: tid=0 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 client.16716 shutdown 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 msgr2=0x561500f49ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40096b0 tx=0x7f9ea401c6d0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 shutdown_connections 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x561500fb8d10 0x561500fbb130 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561500f37140 0x561500f47230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x561500f41a00 0x561500f47840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9e9e000640 0 client.16716 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 --2- 172.21.15.63:0/98809407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561500f36820 0x561500f49ea0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 >> 172.21.15.63:0/98809407 conn(0x561500f32d30 msgr2=0x561500f3f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 shutdown_connections 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 1 -- 172.21.15.63:0/98809407 wait complete. 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.694+0000 7f9ebdbaf740 20 client.16716 trim_cache size 0 max 16384 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:01:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:01:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 msgr2=0x7f9e980a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 0x7f9e980a6300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x561500f37dc0 tx=0x7f9ea4005f90 comp rx=0 tx=0).stop 2024-04-25T19:01:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 msgr2=0x7f9eb41a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0000f80 tx=0x7f9ea0003ba0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 shutdown_connections 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9e980a3e40 0x7f9e980a6300 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9eb4109490 0x7f9eb41a0b60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9eb419a790 0x7f9eb419ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 --2- 172.21.15.63:0/228871066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9eb41043c0 0x7f9eb41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.695+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 >> 172.21.15.63:0/228871066 conn(0x7f9eb40ffc20 msgr2=0x7f9eb4101f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.696+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 shutdown_connections 2024-04-25T19:01:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:17.696+0000 7f9ebae00640 1 -- 172.21.15.63:0/228871066 wait complete. 2024-04-25T19:01:18.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 23.82 MB/sec warmup 76 sec latency 28216.071 ms 2024-04-25T19:01:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:18 smithi114 ceph-mon[32988]: pgmap v464: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:18 smithi063 ceph-mon[28006]: pgmap v464: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:18 smithi070 ceph-mon[32820]: pgmap v464: 97 pgs: 97 active+clean; 414 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 8.5 MiB/s wr, 276 op/s 2024-04-25T19:01:19.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 23.51 MB/sec warmup 77 sec latency 29216.234 ms 2024-04-25T19:01:20.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 23.21 MB/sec warmup 78 sec latency 30216.355 ms 2024-04-25T19:01:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:20 smithi063 ceph-mon[28006]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T19:01:20.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:20 smithi063 ceph-mon[28006]: pgmap v466: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 7.1 MiB/s wr, 252 op/s 2024-04-25T19:01:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:20 smithi070 ceph-mon[32820]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T19:01:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:20 smithi070 ceph-mon[32820]: pgmap v466: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 7.1 MiB/s wr, 252 op/s 2024-04-25T19:01:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:20 smithi114 ceph-mon[32988]: osdmap e117: 12 total, 12 up, 12 in 2024-04-25T19:01:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:20 smithi114 ceph-mon[32988]: pgmap v466: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 7.1 MiB/s wr, 252 op/s 2024-04-25T19:01:21.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 22.92 MB/sec warmup 79 sec latency 31216.511 ms 2024-04-25T19:01:21.522 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:01:22.042 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:01:22.079 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 22.63 MB/sec warmup 80 sec latency 32216.590 ms 2024-04-25T19:01:22.452 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:22 smithi063 ceph-mon[28006]: pgmap v467: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:22 smithi070 ceph-mon[32820]: pgmap v467: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:22 smithi114 ceph-mon[32988]: pgmap v467: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:23.080 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 22.35 MB/sec warmup 81 sec latency 33216.700 ms 2024-04-25T19:01:24.080 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 22.08 MB/sec warmup 82 sec latency 34216.778 ms 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 -- 172.21.15.63:0/3262174071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 msgr2=0x7f2c4410f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 --2- 172.21.15.63:0/3262174071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4410f8b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c009f90 tx=0x7f2c3c00b330 comp rx=0 tx=0).stop 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 -- 172.21.15.63:0/3262174071 shutdown_connections 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 --2- 172.21.15.63:0/3262174071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44108a50 0x7f2c44108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 --2- 172.21.15.63:0/3262174071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4410f8b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 --2- 172.21.15.63:0/3262174071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44102a90 0x7f2c44102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 -- 172.21.15.63:0/3262174071 >> 172.21.15.63:0/3262174071 conn(0x7f2c44079ba0 msgr2=0x7f2c44079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:24.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 -- 172.21.15.63:0/3262174071 shutdown_connections 2024-04-25T19:01:24.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.369+0000 7f2c49400700 1 -- 172.21.15.63:0/3262174071 wait complete. 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.370+0000 7f2c49400700 1 Processor -- start 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.370+0000 7f2c49400700 1 -- start start 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 0x7f2c4419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44108a50 0x7f2c4419fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44112c50 con 0x7f2c44108a50 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44112ad0 con 0x7f2c44103450 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c49400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2c44112dd0 con 0x7f2c44102a90 2024-04-25T19:01:24.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c3be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c3be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41766/0 (socket says 172.21.15.63:41766) 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c3be00700 1 -- 172.21.15.63:0/1422878975 learned_addr learned my addr 172.21.15.63:0/1422878975 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c43200700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44108a50 0x7f2c4419fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.371+0000 7f2c42800700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 0x7f2c4419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 -- 172.21.15.63:0/1422878975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 msgr2=0x7f2c4419b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 -- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44108a50 msgr2=0x7f2c4419fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44108a50 0x7f2c4419fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 -- 172.21.15.63:0/1422878975 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c441a0310 con 0x7f2c44102a90 2024-04-25T19:01:24.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c3be00700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c42800700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 0x7f2c4419b300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400eb90 tx=0x7f2c3400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3400cc90 con 0x7f2c44102a90 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c3400ce30 con 0x7f2c44102a90 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c441a0600 con 0x7f2c44102a90 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.372+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34010430 con 0x7f2c44102a90 2024-04-25T19:01:24.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.373+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c441a7e80 con 0x7f2c44102a90 2024-04-25T19:01:24.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.374+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c44069730 con 0x7f2c44102a90 2024-04-25T19:01:24.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.376+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2c34003ef0 con 0x7f2c44102a90 2024-04-25T19:01:24.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.377+0000 7f2c3b400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 0x7f2c280a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:24.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.377+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f2c340d0a60 con 0x7f2c44102a90 2024-04-25T19:01:24.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.377+0000 7f2c3be00700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 0x7f2c280a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:24.380 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.378+0000 7f2c3be00700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 0x7f2c280a62f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c44101be0 tx=0x7f2c2c009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:24.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.382+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c3409d060 con 0x7f2c44102a90 2024-04-25T19:01:24.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:24 smithi063 ceph-mon[28006]: pgmap v468: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:24.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.664+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c441a8120 con 0x7f2c44102a90 2024-04-25T19:01:24.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.665+0000 7f2c3b400700 1 -- 172.21.15.63:0/1422878975 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11877 (secure 0 0 0) 0x7f2c3409c960 con 0x7f2c44102a90 2024-04-25T19:01:24.668 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:01:24.668 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T18:58:44.004081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 msgr2=0x7f2c280a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 0x7f2c280a62f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c44101be0 tx=0x7f2c2c009260 comp rx=0 tx=0).stop 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 msgr2=0x7f2c4419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 0x7f2c4419b300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2c3400eb90 tx=0x7f2c3400eea0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 shutdown_connections 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2c280a3e30 0x7f2c280a62f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f2c44102a90 0x7f2c4419b300 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c44108a50 0x7f2c4419fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 --2- 172.21.15.63:0/1422878975 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2c44103450 0x7f2c4419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:24.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 >> 172.21.15.63:0/1422878975 conn(0x7f2c44079ba0 msgr2=0x7f2c4410dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:24.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 shutdown_connections 2024-04-25T19:01:24.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:24.668+0000 7f2c49400700 1 -- 172.21.15.63:0/1422878975 wait complete. 2024-04-25T19:01:24.673 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 48 2024-04-25T19:01:24.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:24 smithi070 ceph-mon[32820]: pgmap v468: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:24 smithi114 ceph-mon[32988]: pgmap v468: 97 pgs: 97 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 6.9 MiB/s wr, 248 op/s 2024-04-25T19:01:25.080 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 21.81 MB/sec warmup 83 sec latency 35217.392 ms 2024-04-25T19:01:25.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:25 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1422878975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:01:25.631 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id a638040e --release --await' 2024-04-25T19:01:25.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id a638040e --release --await 2024-04-25T19:01:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:25 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1422878975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:01:25.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:25 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1422878975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:01:26.080 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 21.55 MB/sec warmup 84 sec latency 36217.485 ms 2024-04-25T19:01:26.106 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:01:26.581 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:26 smithi063 ceph-mon[28006]: pgmap v469: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:26 smithi070 ceph-mon[32820]: pgmap v469: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:26 smithi114 ceph-mon[32988]: pgmap v469: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:27.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 21.30 MB/sec warmup 85 sec latency 37217.630 ms 2024-04-25T19:01:28.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30381 21.05 MB/sec warmup 86 sec latency 38217.777 ms 2024-04-25T19:01:28.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.526+0000 7f632ba00700 1 -- 172.21.15.63:0/2109796351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6324108a30 msgr2=0x7f6324108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.526+0000 7f632ba00700 1 --2- 172.21.15.63:0/2109796351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6324108a30 0x7f6324108e10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6314009990 tx=0x7f6314009ca0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 -- 172.21.15.63:0/2109796351 shutdown_connections 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 --2- 172.21.15.63:0/2109796351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f632406b0f0 0x7f632410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 --2- 172.21.15.63:0/2109796351 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6324108a30 0x7f6324108e10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 --2- 172.21.15.63:0/2109796351 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406a750 0x7f632406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 -- 172.21.15.63:0/2109796351 >> 172.21.15.63:0/2109796351 conn(0x7f63240788f0 msgr2=0x7f6324078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 -- 172.21.15.63:0/2109796351 shutdown_connections 2024-04-25T19:01:28.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.527+0000 7f632ba00700 1 -- 172.21.15.63:0/2109796351 wait complete. 2024-04-25T19:01:28.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.528+0000 7f632ba00700 1 Processor -- start 2024-04-25T19:01:28.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.528+0000 7f632ba00700 1 -- start start 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406b0f0 0x7f6324101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6324108a30 0x7f6324102490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112c00 con 0x7f6324108a30 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112a80 con 0x7f632406a750 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f632ba00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f6324112d80 con 0x7f632406b0f0 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f6329000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f6329000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54150/0 (socket says 172.21.15.63:54150) 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f6329000700 1 -- 172.21.15.63:0/540310905 learned_addr learned my addr 172.21.15.63:0/540310905 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f631be00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406b0f0 0x7f6324101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.529+0000 7f6329a00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6324108a30 0x7f6324102490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 -- 172.21.15.63:0/540310905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406b0f0 msgr2=0x7f6324101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406b0f0 0x7f6324101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 -- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6324108a30 msgr2=0x7f6324102490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6324108a30 0x7f6324102490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 -- 172.21.15.63:0/540310905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63241a76e0 con 0x7f632406a750 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.530+0000 7f6329000700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6314000c00 tx=0x7f63140057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.531+0000 7f6322000700 1 -- 172.21.15.63:0/540310905 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314027070 con 0x7f632406a750 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.531+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63241a7970 con 0x7f632406a750 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.531+0000 7f6322000700 1 -- 172.21.15.63:0/540310905 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6314022470 con 0x7f632406a750 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.531+0000 7f6322000700 1 -- 172.21.15.63:0/540310905 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314017650 con 0x7f632406a750 2024-04-25T19:01:28.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.531+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63241a7e20 con 0x7f632406a750 2024-04-25T19:01:28.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.535+0000 7f6322000700 1 -- 172.21.15.63:0/540310905 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f631402a070 con 0x7f632406a750 2024-04-25T19:01:28.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 Processor -- start 2024-04-25T19:01:28.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 -- start start 2024-04-25T19:01:28.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 0x562bc503cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc504a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562bc4f8ff90 con 0x562bc503c920 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562bc4f36830 con 0x562bc50480f0 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.536+0000 7f6336309b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562bc4f8bf50 con 0x562bc503d240 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6322000700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 0x7f63080a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 0x562bc503cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6322000700 1 -- 172.21.15.63:0/540310905 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f63140d14f0 con 0x7f632406a750 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f631be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc504a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47478/0 (socket says 172.21.15.63:47478) 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 0x562bc503cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60866/0 (socket says 172.21.15.63:60866) 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 -- 172.21.15.63:0/1339429064 learned_addr learned my addr 172.21.15.63:0/1339429064 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:28.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329a00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 0x7f63080a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 -- 172.21.15.63:0/1339429064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 msgr2=0x562bc504a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc504a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 -- 172.21.15.63:0/1339429064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 msgr2=0x562bc503cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 0x562bc503cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f6329000700 1 -- 172.21.15.63:0/1339429064 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bc4f2c780 con 0x562bc503d240 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.537+0000 7f631be00700 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc504a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.538+0000 7f6329000700 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6314006010 tx=0x7f6314099120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=503488b69126411f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.538+0000 7f6329a00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 0x7f63080a62f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f631c009fd0 tx=0x7f631c00b040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.538+0000 7f631a000700 1 -- 172.21.15.63:0/1339429064 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631409cde0 con 0x562bc503d240 2024-04-25T19:01:28.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.538+0000 7f631a000700 1 -- 172.21.15.63:0/1339429064 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631409db80 con 0x562bc503d240 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.538+0000 7f631a000700 1 -- 172.21.15.63:0/1339429064 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63140cee20 con 0x562bc503d240 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 -- 172.21.15.63:0/1339429064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 msgr2=0x562bc5047bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6314006010 tx=0x7f6314099120 comp rx=0 tx=0).stop 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 -- 172.21.15.63:0/1339429064 shutdown_connections 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc504a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503c920 0x562bc503cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 --2- 172.21.15.63:0/1339429064 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503d240 0x562bc5047bb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 -- 172.21.15.63:0/1339429064 >> 172.21.15.63:0/1339429064 conn(0x562bc4f6b6a0 msgr2=0x562bc5038dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 -- 172.21.15.63:0/1339429064 shutdown_connections 2024-04-25T19:01:28.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.539+0000 7f6336309b80 1 -- 172.21.15.63:0/1339429064 wait complete. 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.540+0000 7f6336309b80 1 -- start start 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.540+0000 7f6336309b80 1 Processor -- start 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50477e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc5044b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562bc504aa20 con 0x562bc503d240 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562bc4f49960 con 0x562bc50480f0 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x562bc504aba0 con 0x562bc503c920 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6336309b80 10 client.0 _ll_register_callbacks cb 0x562bc4f28820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6329a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6329a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54170/0 (socket says 172.21.15.63:54170) 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f6329a00700 1 -- 172.21.15.63:0/593581651 learned_addr learned my addr 172.21.15.63:0/593581651 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.541+0000 7f631be00700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc5044b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 -- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 msgr2=0x562bc50450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 -- 172.21.15.63:0/593581651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 msgr2=0x562bc5044b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc5044b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 -- 172.21.15.63:0/593581651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bc4edd9d0 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329a00700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6329000700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50477e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f63140a6220 tx=0x7f63140cf050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=995ab694bfa96eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6312a00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63140b5020 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f631409d8f0 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.542+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f631409db80 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f631409cde0 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 10 client.26947 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562bc4f26bf0 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631409db80 con 0x562bc503c920 2024-04-25T19:01:28.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f63140d0070 con 0x562bc503c920 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 10 client.26947 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 10 client.26947 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 20 client.26947 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc5057500 0x562bc50598f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 4 client.26947 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.543+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bc4f9d780 con 0x562bc5057500 2024-04-25T19:01:28.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.544+0000 7f631be00700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc5057500 0x562bc50598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.544+0000 7f631be00700 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc5057500 0x562bc50598f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312ebfa523958a71 server_cookie=c7e06f379fccea0e in_seq=0 out_seq=0 2024-04-25T19:01:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.544+0000 7f6312a00700 10 client.26947 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:28.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.545+0000 7f6312a00700 1 -- 172.21.15.63:0/593581651 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bc4f9d780 con 0x562bc5057500 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.545+0000 7f6312a00700 10 client.26947 handle_command_reply: tid=0 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.545+0000 7f6336309b80 1 client.26947 shutdown 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 msgr2=0x562bc50477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50477e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f63140a6220 tx=0x7f63140cf050 comp rx=0 tx=0).stop 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 shutdown_connections 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc5057500 0x562bc50598f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50477e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc50450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6312a00700 0 client.26947 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 --2- 172.21.15.63:0/593581651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc5044b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 >> 172.21.15.63:0/593581651 conn(0x562bc4f6b6a0 msgr2=0x562bc50389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 shutdown_connections 2024-04-25T19:01:28.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 1 -- 172.21.15.63:0/593581651 wait complete. 2024-04-25T19:01:28.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.546+0000 7f6336309b80 20 client.26947 trim_cache size 0 max 16384 2024-04-25T19:01:28.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 Processor -- start 2024-04-25T19:01:28.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 -- start start 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc50401a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f63241b1e30 con 0x562bc503d240 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63241af600 con 0x562bc50480f0 2024-04-25T19:01:28.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.568+0000 7f6336309b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63241ad450 con 0x562bc503c920 2024-04-25T19:01:28.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f631be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc50401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54178/0 (socket says 172.21.15.63:54178) 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 -- 172.21.15.63:0/1782509312 learned_addr learned my addr 172.21.15.63:0/1782509312 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 -- 172.21.15.63:0/1782509312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 msgr2=0x562bc50538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.569+0000 7f6329a00700 1 -- 172.21.15.63:0/1782509312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 msgr2=0x562bc50401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6329a00700 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc50401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6329a00700 1 -- 172.21.15.63:0/1782509312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bc503d070 con 0x562bc50480f0 2024-04-25T19:01:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f631be00700 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc50401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6329a00700 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f631c012410 tx=0x7f631c00f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ecc2fe4a388cae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6319600700 1 -- 172.21.15.63:0/1782509312 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c004bb0 con 0x562bc50480f0 2024-04-25T19:01:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6319600700 1 -- 172.21.15.63:0/1782509312 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631c004d50 con 0x562bc50480f0 2024-04-25T19:01:28.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.570+0000 7f6319600700 1 -- 172.21.15.63:0/1782509312 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631c01a8a0 con 0x562bc50480f0 2024-04-25T19:01:28.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 -- 172.21.15.63:0/1782509312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 msgr2=0x562bc505f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f631c012410 tx=0x7f631c00f000 comp rx=0 tx=0).stop 2024-04-25T19:01:28.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 -- 172.21.15.63:0/1782509312 shutdown_connections 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc50538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc50480f0 0x562bc505f080 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 --2- 172.21.15.63:0/1782509312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc503d240 0x562bc50401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 -- 172.21.15.63:0/1782509312 >> 172.21.15.63:0/1782509312 conn(0x562bc4f6b6a0 msgr2=0x562bc5043c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 -- 172.21.15.63:0/1782509312 shutdown_connections 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.571+0000 7f6336309b80 1 -- 172.21.15.63:0/1782509312 wait complete. 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 -- start start 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 Processor -- start 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc503d240 0x562bc50424d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc50480f0 0x562bc50bc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f63241adac0 con 0x562bc50480f0 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63241aef30 con 0x562bc503d240 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.572+0000 7f6336309b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63241ae190 con 0x562bc503c920 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6336309b80 10 client.0 _ll_register_callbacks cb 0x562bc4f28820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47490/0 (socket says 172.21.15.63:47490) 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 -- 172.21.15.63:0/3169202842 learned_addr learned my addr 172.21.15.63:0/3169202842 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329a00700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc50480f0 0x562bc50bc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f631be00700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc503d240 0x562bc50424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 -- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc503d240 msgr2=0x562bc50424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc503d240 0x562bc50424d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 -- 172.21.15.63:0/3169202842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc50480f0 msgr2=0x562bc50bc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc50480f0 0x562bc50bc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.573+0000 7f6329000700 1 -- 172.21.15.63:0/3169202842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bc50bce30 con 0x562bc503c920 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6329000700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63140a5cc0 tx=0x7f63140990f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3847b5bcf6bafca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6312a00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63140b5070 con 0x562bc503c920 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f631c004d50 con 0x562bc503c920 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f63140bc170 con 0x562bc503c920 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f63140a6060 con 0x562bc503c920 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6336309b80 10 client.26953 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:28.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562bc50bce30 con 0x562bc503c920 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.574+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f631c004d50 con 0x562bc503c920 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f6314085070 con 0x562bc503c920 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 10 client.26953 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 10 client.26953 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 20 client.26953 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc50be3d0 0x562bc50c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 4 client.26953 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "a638040e", "release": true, "await": true}] 2024-04-25T19:01:28.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "a638040e", "release": true, "await": true}) -- 0x562bc4f97820 con 0x562bc50be3d0 2024-04-25T19:01:28.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.575+0000 7f631be00700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc50be3d0 0x562bc50c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:28.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.576+0000 7f631be00700 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc50be3d0 0x562bc50c07c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3336f493ed0b269d server_cookie=83419479fff1ab72 in_seq=0 out_seq=0 2024-04-25T19:01:28.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.576+0000 7f6312a00700 10 client.26953 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:28.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.606+0000 7f6312a00700 1 -- 172.21.15.63:0/3169202842 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x562bc4f97820 con 0x562bc50be3d0 2024-04-25T19:01:28.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.606+0000 7f6312a00700 10 client.26953 handle_command_reply: tid=0 2024-04-25T19:01:28.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.606+0000 7f6336309b80 1 client.26953 shutdown 2024-04-25T19:01:28.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.606+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 msgr2=0x562bc5041f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.606+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f63140a5cc0 tx=0x7f63140990f0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.610 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:28 smithi063 ceph-mon[28006]: pgmap v470: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:28.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 shutdown_connections 2024-04-25T19:01:28.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x562bc50be3d0 0x562bc50c07c0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x562bc503c920 0x562bc5041f90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562bc50480f0 0x562bc50bc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 --2- 172.21.15.63:0/3169202842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bc503d240 0x562bc50424d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 >> 172.21.15.63:0/3169202842 conn(0x562bc4f6b6a0 msgr2=0x562bc5043c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6312a00700 0 client.26953 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 shutdown_connections 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 1 -- 172.21.15.63:0/3169202842 wait complete. 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.607+0000 7f6336309b80 20 client.26953 trim_cache size 0 max 16384 2024-04-25T19:01:28.611 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 48, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 44, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "a638040e": { 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 44, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 90.3, 2024-04-25T19:01:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:01:28.613 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 msgr2=0x7f63080a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 0x7f63080a62f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f631c009fd0 tx=0x7f631c00b040 comp rx=0 tx=0).stop 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 msgr2=0x7f6324101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6314000c00 tx=0x7f63140057c0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 shutdown_connections 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f63080a3e30 0x7f63080a62f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6324108a30 0x7f6324102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f632406a750 0x7f6324101a10 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 --2- 172.21.15.63:0/540310905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f632406b0f0 0x7f6324101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:28.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.608+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 >> 172.21.15.63:0/540310905 conn(0x7f63240788f0 msgr2=0x7f63240ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:28.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.609+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 shutdown_connections 2024-04-25T19:01:28.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:01:28.609+0000 7f632ba00700 1 -- 172.21.15.63:0/540310905 wait complete. 2024-04-25T19:01:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:28 smithi070 ceph-mon[32820]: pgmap v470: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:28 smithi114 ceph-mon[32988]: pgmap v470: 97 pgs: 97 active+clean; 426 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 5.3 MiB/s wr, 213 op/s 2024-04-25T19:01:29.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 30606 20.97 MB/sec warmup 87 sec latency 38772.612 ms 2024-04-25T19:01:29.492 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'a638040e', seconds elapsed: 47.9 2024-04-25T19:01:30.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 31188 21.13 MB/sec warmup 88 sec latency 1371.291 ms 2024-04-25T19:01:30.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:30 smithi070 ceph-mon[32820]: pgmap v471: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 169 op/s 2024-04-25T19:01:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:30 smithi114 ceph-mon[32988]: pgmap v471: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 169 op/s 2024-04-25T19:01:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 -- 172.21.15.63:0/1471982768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc104d90 msgr2=0x7fddcc108dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 --2- 172.21.15.63:0/1471982768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc104d90 0x7fddcc108dd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fddb8009900 tx=0x7fddb802a3f0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 -- 172.21.15.63:0/1471982768 shutdown_connections 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 --2- 172.21.15.63:0/1471982768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc104d90 0x7fddcc108dd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 --2- 172.21.15.63:0/1471982768 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc1094b0 0x7fddcc111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 --2- 172.21.15.63:0/1471982768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc1047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.864+0000 7fddd1e00640 1 -- 172.21.15.63:0/1471982768 >> 172.21.15.63:0/1471982768 conn(0x7fddcc0ffc60 msgr2=0x7fddcc102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:30.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.865+0000 7fddd1e00640 1 -- 172.21.15.63:0/1471982768 shutdown_connections 2024-04-25T19:01:30.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.865+0000 7fddd1e00640 1 -- 172.21.15.63:0/1471982768 wait complete. 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 Processor -- start 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 -- start start 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc19ad90 0x7fddcc19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc078c20 con 0x7fddcc1043e0 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc078aa0 con 0x7fddcc1094b0 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.866+0000 7fddd1e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc078da0 con 0x7fddcc19ad90 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54208/0 (socket says 172.21.15.63:54208) 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 -- 172.21.15.63:0/362852155 learned_addr learned my addr 172.21.15.63:0/362852155 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddcb400640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc19ad90 0x7fddcc19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddcaa00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 -- 172.21.15.63:0/362852155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc19ad90 msgr2=0x7fddcc19b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc19ad90 0x7fddcc19b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 -- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 msgr2=0x7fddcc196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddca000640 1 -- 172.21.15.63:0/362852155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc19b880 con 0x7fddcc1094b0 2024-04-25T19:01:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.867+0000 7fddcaa00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.868+0000 7fddca000640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fddb8005ab0 tx=0x7fddb8003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.868+0000 7fddd0a00640 1 -- 172.21.15.63:0/362852155 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb803f070 con 0x7fddcc1094b0 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.868+0000 7fddd0a00640 1 -- 172.21.15.63:0/362852155 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddb8031b50 con 0x7fddcc1094b0 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.868+0000 7fddd0a00640 1 -- 172.21.15.63:0/362852155 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8035030 con 0x7fddcc1094b0 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.868+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc19bb10 con 0x7fddcc1094b0 2024-04-25T19:01:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.869+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc1a3280 con 0x7fddcc1094b0 2024-04-25T19:01:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.873+0000 7fddd4c60740 1 Processor -- start 2024-04-25T19:01:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.873+0000 7fddd4c60740 1 -- start start 2024-04-25T19:01:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.873+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.873+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cd24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.874+0000 7fddcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.874+0000 7fddd0a00640 1 -- 172.21.15.63:0/362852155 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fddb8042070 con 0x7fddcc1094b0 2024-04-25T19:01:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.874+0000 7fddcb400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54222/0 (socket says 172.21.15.63:54222) 2024-04-25T19:01:30.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:30 smithi063 ceph-mon[28006]: pgmap v471: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.9 MiB/s wr, 169 op/s 2024-04-25T19:01:30.879 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.876+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 0x55f869cd4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.877+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f869c2d900 con 0x55f869cc8140 2024-04-25T19:01:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.877+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f869bd3ef0 con 0x55f869cc7820 2024-04-25T19:01:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.877+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f869c298c0 con 0x55f869cd2a00 2024-04-25T19:01:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.876+0000 7fddcb400640 1 -- 172.21.15.63:0/3209765339 learned_addr learned my addr 172.21.15.63:0/3209765339 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:30.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.877+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cd24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.878+0000 7fddd0a00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 0x7fddb00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.881 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.878+0000 7fddca000640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 0x55f869cd4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.882 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcaa00640 1 -- 172.21.15.63:0/3209765339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 msgr2=0x55f869cd4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 0x55f869cd4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcaa00640 1 -- 172.21.15.63:0/3209765339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 msgr2=0x55f869cc7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcaa00640 1 -- 172.21.15.63:0/3209765339 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f869940210 con 0x55f869cc8140 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddcb400640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.879+0000 7fddd0a00640 1 -- 172.21.15.63:0/362852155 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7fddb80f0e80 con 0x7fddcc1094b0 2024-04-25T19:01:30.883 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.880+0000 7fddca000640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 0x55f869cd4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.881+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cd24c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009c30 tx=0x7fddbc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19d03917039e40f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.881+0000 7fddcb400640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 0x7fddb00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.882+0000 7fddb7e00640 1 -- 172.21.15.63:0/3209765339 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc012450 con 0x55f869cc8140 2024-04-25T19:01:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.883+0000 7fddb7e00640 1 -- 172.21.15.63:0/3209765339 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddbc012aa0 con 0x55f869cc8140 2024-04-25T19:01:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.883+0000 7fddb7e00640 1 -- 172.21.15.63:0/3209765339 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc00fa60 con 0x55f869cc8140 2024-04-25T19:01:30.888 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.886+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 msgr2=0x55f869cd24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.889 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.886+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cd24c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009c30 tx=0x7fddbc0117b0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddcb400640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 0x7fddb00a6380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fddc4013d90 tx=0x7fddc4004ab0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 shutdown_connections 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cd2a00 0x55f869cd4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cd24c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3209765339 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cc7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 >> 172.21.15.63:0/3209765339 conn(0x55f869cc3d30 msgr2=0x55f869cc4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.888+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 shutdown_connections 2024-04-25T19:01:30.891 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.889+0000 7fddd4c60740 1 -- 172.21.15.63:0/3209765339 wait complete. 2024-04-25T19:01:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.889+0000 7fddd4c60740 1 -- start start 2024-04-25T19:01:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 Processor -- start 2024-04-25T19:01:30.892 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869ccf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f869cd5630 con 0x55f869cd2a00 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f869be7310 con 0x55f869cc7820 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.890+0000 7fddd4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f869cd57b0 con 0x55f869cc8140 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddd4c60740 10 client.0 _ll_register_callbacks cb 0x55f869bb5cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47534/0 (socket says 172.21.15.63:47534) 2024-04-25T19:01:30.893 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 -- 172.21.15.63:0/1790757642 learned_addr learned my addr 172.21.15.63:0/1790757642 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddcaa00640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869ccf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 -- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 msgr2=0x55f869ccf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869ccf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 -- 172.21.15.63:0/1790757642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 msgr2=0x55f869cd0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.894 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.891+0000 7fddca000640 1 -- 172.21.15.63:0/1790757642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f869a0c620 con 0x55f869cc8140 2024-04-25T19:01:30.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.892+0000 7fddca000640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fddb80bcd10 tx=0x7fddb8003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5558bf6f15d45b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.895 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.892+0000 7fddb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8049070 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7fddb80ee7b0 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddbc012aa0 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fddb80c1130 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddd4c60740 10 client.26965 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55f869cd5690 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.893+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddbc012aa0 con 0x55f869cc8140 2024-04-25T19:01:30.896 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7fddb80d6140 con 0x55f869cc8140 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 10 client.26965 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 10 client.26965 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 20 client.26965 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869ce2cc0 0x55f869ce5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 4 client.26965 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:01:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.894+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f869c418b0 con 0x55f869ce2cc0 2024-04-25T19:01:30.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.897+0000 7fddcaa00640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869ce2cc0 0x55f869ce5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.898+0000 7fddcaa00640 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869ce2cc0 0x55f869ce5120 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312588fee9f4cc6c server_cookie=10667065a920c050 in_seq=0 out_seq=0 2024-04-25T19:01:30.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.898+0000 7fddb4c00640 10 client.26965 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.903+0000 7fddb4c00640 1 -- 172.21.15.63:0/1790757642 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f869c418b0 con 0x55f869ce2cc0 2024-04-25T19:01:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.903+0000 7fddb4c00640 10 client.26965 handle_command_reply: tid=0 2024-04-25T19:01:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.904+0000 7fddd4c60740 1 client.26965 shutdown 2024-04-25T19:01:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.904+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 msgr2=0x55f869ccfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.904+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fddb80bcd10 tx=0x7fddb8003f90 comp rx=0 tx=0).stop 2024-04-25T19:01:30.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 shutdown_connections 2024-04-25T19:01:30.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869ce2cc0 0x55f869ce5120 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869ccfe40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 --2- 172.21.15.63:0/1790757642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869ccf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 >> 172.21.15.63:0/1790757642 conn(0x55f869cc3d30 msgr2=0x55f869cc9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:30.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddb4c00640 0 client.26965 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.905+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 shutdown_connections 2024-04-25T19:01:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.906+0000 7fddd4c60740 1 -- 172.21.15.63:0/1790757642 wait complete. 2024-04-25T19:01:30.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.906+0000 7fddd4c60740 20 client.26965 trim_cache size 0 max 16384 2024-04-25T19:01:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.941+0000 7fddd4c60740 1 Processor -- start 2024-04-25T19:01:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 -- start start 2024-04-25T19:01:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cd2a00 0x55f869ccffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1ae2b0 con 0x55f869cc8140 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1abac0 con 0x55f869cd2a00 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddd4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1a9930 con 0x55f869cc7820 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddcaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddcaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47546/0 (socket says 172.21.15.63:47546) 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.942+0000 7fddcaa00640 1 -- 172.21.15.63:0/3270609082 learned_addr learned my addr 172.21.15.63:0/3270609082 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcb400640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cd2a00 0x55f869ccffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddca000640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcaa00640 1 -- 172.21.15.63:0/3270609082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cd2a00 msgr2=0x55f869ccffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cd2a00 0x55f869ccffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcaa00640 1 -- 172.21.15.63:0/3270609082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 msgr2=0x55f869cca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddcaa00640 1 -- 172.21.15.63:0/3270609082 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f869cbf690 con 0x55f869cc7820 2024-04-25T19:01:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.943+0000 7fddca000640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:01:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.944+0000 7fddcaa00640 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009c30 tx=0x7fddbc009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cbfdfb67ef37a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.944+0000 7fddb7400640 1 -- 172.21.15.63:0/3270609082 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc009e40 con 0x55f869cc7820 2024-04-25T19:01:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.944+0000 7fddb7400640 1 -- 172.21.15.63:0/3270609082 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddbc0054f0 con 0x55f869cc7820 2024-04-25T19:01:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.944+0000 7fddb7400640 1 -- 172.21.15.63:0/3270609082 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc0035f0 con 0x55f869cc7820 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 -- 172.21.15.63:0/3270609082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 msgr2=0x55f869cde4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009c30 tx=0x7fddbc009820 comp rx=0 tx=0).stop 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 -- 172.21.15.63:0/3270609082 shutdown_connections 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cd2a00 0x55f869ccffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cc8140 0x55f869cca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 --2- 172.21.15.63:0/3270609082 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc7820 0x55f869cde4f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 -- 172.21.15.63:0/3270609082 >> 172.21.15.63:0/3270609082 conn(0x55f869cc3d30 msgr2=0x55f869ccf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 -- 172.21.15.63:0/3270609082 shutdown_connections 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.945+0000 7fddd4c60740 1 -- 172.21.15.63:0/3270609082 wait complete. 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.946+0000 7fddd4c60740 1 -- start start 2024-04-25T19:01:30.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.946+0000 7fddd4c60740 1 Processor -- start 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869cd8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1a9f80 con 0x55f869cd2a00 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1ab3f0 con 0x55f869cc7820 2024-04-25T19:01:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1aa650 con 0x55f869cc8140 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddd4c60740 10 client.0 _ll_register_callbacks cb 0x7fddb8005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddcaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddcaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:54248/0 (socket says 172.21.15.63:54248) 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddcaa00640 1 -- 172.21.15.63:0/4057401045 learned_addr learned my addr 172.21.15.63:0/4057401045 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddca000640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869cd8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddcb400640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.948+0000 7fddcaa00640 1 -- 172.21.15.63:0/4057401045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 msgr2=0x55f869cd8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddcaa00640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869cd8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddcaa00640 1 -- 172.21.15.63:0/4057401045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 msgr2=0x55f869cd8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddcaa00640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddcaa00640 1 -- 172.21.15.63:0/4057401045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f869cd8e50 con 0x55f869cc7820 2024-04-25T19:01:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddcaa00640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006010 tx=0x7fddbc009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53198b2897994f0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.949+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc0052a0 con 0x55f869cc7820 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7fddbc02a1e0 con 0x55f869cc7820 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddbc0054f0 con 0x55f869cc7820 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fddbc02a420 con 0x55f869cc7820 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddd4c60740 10 client.26957 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55f869d491c0 con 0x55f869cc7820 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddbc0054f0 con 0x55f869cc7820 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.950+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7fddbc043510 con 0x55f869cc7820 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 10 client.26957 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 10 client.26957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 20 client.26957 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869d49660 0x55f869d4ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 4 client.26957 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f869c3b320 con 0x55f869d49660 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddca000640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869d49660 0x55f869d4ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddca000640 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869d49660 0x55f869d4ba80 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41e741fae39d8c1b server_cookie=b2a0c836e8f0c951 in_seq=0 out_seq=0 2024-04-25T19:01:30.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:30.951+0000 7fddb4c00640 10 client.26957 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:31.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 31873 21.11 MB/sec warmup 89 sec latency 2371.367 ms 2024-04-25T19:01:32.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 32670 21.53 MB/sec warmup 90 sec latency 3371.429 ms 2024-04-25T19:01:32.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:32 smithi070 ceph-mon[32820]: pgmap v472: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:32 smithi114 ceph-mon[32988]: pgmap v472: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:32.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:32 smithi063 ceph-mon[28006]: pgmap v472: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:33.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 33313 21.70 MB/sec warmup 91 sec latency 4371.509 ms 2024-04-25T19:01:34.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 33831 21.72 MB/sec warmup 92 sec latency 5371.597 ms 2024-04-25T19:01:34.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:34 smithi070 ceph-mon[32820]: pgmap v473: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:34 smithi114 ceph-mon[32988]: pgmap v473: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:34.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:34 smithi063 ceph-mon[28006]: pgmap v473: 97 pgs: 97 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-04-25T19:01:35.081 INFO:tasks.workunit.client.0.smithi063.stdout: 10 34385 21.75 MB/sec warmup 93 sec latency 6371.741 ms 2024-04-25T19:01:36.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 35236 21.96 MB/sec warmup 94 sec latency 7371.981 ms 2024-04-25T19:01:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:36 smithi070 ceph-mon[32820]: pgmap v474: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.5 MiB/s wr, 365 op/s 2024-04-25T19:01:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:36 smithi114 ceph-mon[32988]: pgmap v474: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.5 MiB/s wr, 365 op/s 2024-04-25T19:01:36.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:36 smithi063 ceph-mon[28006]: pgmap v474: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.5 MiB/s wr, 365 op/s 2024-04-25T19:01:37.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 36078 22.36 MB/sec warmup 95 sec latency 8372.068 ms 2024-04-25T19:01:38.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 36605 22.44 MB/sec warmup 96 sec latency 9372.178 ms 2024-04-25T19:01:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:38 smithi070 ceph-mon[32820]: pgmap v475: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-04-25T19:01:38.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:38 smithi114 ceph-mon[32988]: pgmap v475: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-04-25T19:01:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:38 smithi063 ceph-mon[28006]: pgmap v475: 97 pgs: 97 active+clean; 449 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-04-25T19:01:39.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 37082 22.39 MB/sec warmup 97 sec latency 10372.307 ms 2024-04-25T19:01:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.190+0000 7fddb4c00640 1 -- 172.21.15.63:0/4057401045 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f869c3b320 con 0x55f869d49660 2024-04-25T19:01:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.190+0000 7fddb4c00640 10 client.26957 handle_command_reply: tid=0 2024-04-25T19:01:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.190+0000 7fddd4c60740 1 client.26957 shutdown 2024-04-25T19:01:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 msgr2=0x55f869cdadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:39.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fddbc006010 tx=0x7fddbc009ee0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 shutdown_connections 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f869d49660 0x55f869d4ba80 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f869cd2a00 0x55f869cd8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f869cc8140 0x55f869cd8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 --2- 172.21.15.63:0/4057401045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f869cc7820 0x55f869cdadc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddb4c00640 0 client.26957 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:39.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.191+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 >> 172.21.15.63:0/4057401045 conn(0x55f869cc3d30 msgr2=0x55f869cd0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.192+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 shutdown_connections 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.192+0000 7fddd4c60740 1 -- 172.21.15.63:0/4057401045 wait complete. 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.192+0000 7fddd4c60740 20 client.26957 trim_cache size 0 max 16384 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:01:39.194 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 msgr2=0x7fddb00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 0x7fddb00a6380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fddc4013d90 tx=0x7fddc4004ab0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 msgr2=0x7fddcc197000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fddb8005ab0 tx=0x7fddb8003b90 comp rx=0 tx=0).stop 2024-04-25T19:01:39.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 shutdown_connections 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fddb00a3ec0 0x7fddb00a6380 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fddcc1094b0 0x7fddcc197000 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fddcc19ad90 0x7fddcc19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 --2- 172.21.15.63:0/362852155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddcc1043e0 0x7fddcc196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 >> 172.21.15.63:0/362852155 conn(0x7fddcc0ffc60 msgr2=0x7fddcc107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 shutdown_connections 2024-04-25T19:01:39.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:39.193+0000 7fddd1e00640 1 -- 172.21.15.63:0/362852155 wait complete. 2024-04-25T19:01:40.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 37767 22.50 MB/sec warmup 98 sec latency 10495.676 ms 2024-04-25T19:01:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:40 smithi070 ceph-mon[32820]: pgmap v476: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-04-25T19:01:40.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:40 smithi114 ceph-mon[32988]: pgmap v476: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-04-25T19:01:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:40 smithi063 ceph-mon[28006]: pgmap v476: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-04-25T19:01:41.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 38587 22.63 MB/sec warmup 99 sec latency 76.838 ms 2024-04-25T19:01:42.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 39329 22.92 MB/sec warmup 100 sec latency 118.260 ms 2024-04-25T19:01:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:42 smithi070 ceph-mon[32820]: pgmap v477: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:42.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:42 smithi114 ceph-mon[32988]: pgmap v477: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:42.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:42 smithi063 ceph-mon[28006]: pgmap v477: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:43.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 39963 23.12 MB/sec warmup 101 sec latency 39.055 ms 2024-04-25T19:01:44.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 40523 23.13 MB/sec warmup 102 sec latency 55.456 ms 2024-04-25T19:01:44.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:44 smithi070 ceph-mon[32820]: pgmap v478: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:44 smithi114 ceph-mon[32988]: pgmap v478: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:44.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:44 smithi063 ceph-mon[28006]: pgmap v478: 97 pgs: 97 active+clean; 472 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.5 MiB/s wr, 454 op/s 2024-04-25T19:01:45.082 INFO:tasks.workunit.client.0.smithi063.stdout: 10 41098 23.16 MB/sec warmup 103 sec latency 68.359 ms 2024-04-25T19:01:46.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 41800 23.32 MB/sec warmup 104 sec latency 51.111 ms 2024-04-25T19:01:46.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:46 smithi070 ceph-mon[32820]: pgmap v479: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 661 op/s 2024-04-25T19:01:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:46 smithi114 ceph-mon[32988]: pgmap v479: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 661 op/s 2024-04-25T19:01:46.881 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:46 smithi063 ceph-mon[28006]: pgmap v479: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 7.2 MiB/s wr, 661 op/s 2024-04-25T19:01:47.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 42503 23.42 MB/sec warmup 105 sec latency 52.327 ms 2024-04-25T19:01:48.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 43203 23.61 MB/sec warmup 106 sec latency 46.720 ms 2024-04-25T19:01:48.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:48 smithi070 ceph-mon[32820]: pgmap v480: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 5.0 MiB/s wr, 403 op/s 2024-04-25T19:01:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:48 smithi114 ceph-mon[32988]: pgmap v480: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 5.0 MiB/s wr, 403 op/s 2024-04-25T19:01:48.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:48 smithi063 ceph-mon[28006]: pgmap v480: 97 pgs: 97 active+clean; 498 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 5.0 MiB/s wr, 403 op/s 2024-04-25T19:01:49.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 43789 23.73 MB/sec warmup 107 sec latency 81.880 ms 2024-04-25T19:01:50.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 44384 23.77 MB/sec warmup 108 sec latency 97.385 ms 2024-04-25T19:01:50.698 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:50 smithi063 ceph-mon[28006]: pgmap v481: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 599 op/s 2024-04-25T19:01:50.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:50 smithi070 ceph-mon[32820]: pgmap v481: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 599 op/s 2024-04-25T19:01:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:50 smithi114 ceph-mon[32988]: pgmap v481: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 599 op/s 2024-04-25T19:01:51.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 45000 23.79 MB/sec warmup 109 sec latency 91.809 ms 2024-04-25T19:01:52.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 45767 24.01 MB/sec warmup 110 sec latency 44.539 ms 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.350+0000 7f941b600640 1 -- 172.21.15.63:0/318689107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 msgr2=0x7f9414103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.350+0000 7f941b600640 1 --2- 172.21.15.63:0/318689107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f9414103440 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9408009900 tx=0x7f940802a480 comp rx=0 tx=0).stop 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.351+0000 7f941b600640 1 -- 172.21.15.63:0/318689107 shutdown_connections 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.351+0000 7f941b600640 1 --2- 172.21.15.63:0/318689107 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 0x7f941410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.351+0000 7f941b600640 1 --2- 172.21.15.63:0/318689107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f941410c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.351+0000 7f941b600640 1 --2- 172.21.15.63:0/318689107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f9414103440 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.351+0000 7f941b600640 1 -- 172.21.15.63:0/318689107 >> 172.21.15.63:0/318689107 conn(0x7f9414077fd0 msgr2=0x7f94140783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.352+0000 7f941b600640 1 -- 172.21.15.63:0/318689107 shutdown_connections 2024-04-25T19:01:52.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.352+0000 7f941b600640 1 -- 172.21.15.63:0/318689107 wait complete. 2024-04-25T19:01:52.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.353+0000 7f941b600640 1 Processor -- start 2024-04-25T19:01:52.354 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.353+0000 7f941b600640 1 -- start start 2024-04-25T19:01:52.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f94141a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 0x7f941410b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9414112200 con 0x7f9414103980 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9414112080 con 0x7f9414103060 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f941b600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9414112380 con 0x7f941410c580 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56172/0 (socket says 172.21.15.63:56172) 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 -- 172.21.15.63:0/1854506271 learned_addr learned my addr 172.21.15.63:0/1854506271 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9413e00640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f94141a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9419600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 0x7f941410b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 -- 172.21.15.63:0/1854506271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 msgr2=0x7f941410b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 0x7f941410b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 -- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 msgr2=0x7f94141a5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f94141a5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9418c00640 1 -- 172.21.15.63:0/1854506271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941410ba50 con 0x7f9414103060 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.354+0000 7f9413e00640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f94141a5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f9418c00640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9408005ab0 tx=0x7f9408003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f9411600640 1 -- 172.21.15.63:0/1854506271 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940803f070 con 0x7f9414103060 2024-04-25T19:01:52.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f9411600640 1 -- 172.21.15.63:0/1854506271 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9408031d60 con 0x7f9414103060 2024-04-25T19:01:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f9411600640 1 -- 172.21.15.63:0/1854506271 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408035030 con 0x7f9414103060 2024-04-25T19:01:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941410bce0 con 0x7f9414103060 2024-04-25T19:01:52.358 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.355+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94141ab9b0 con 0x7f9414103060 2024-04-25T19:01:52.360 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.359+0000 7f9411600640 1 -- 172.21.15.63:0/1854506271 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9408042070 con 0x7f9414103060 2024-04-25T19:01:52.362 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.360+0000 7f941e5d8740 1 Processor -- start 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.360+0000 7f941e5d8740 1 -- start start 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f9411600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 0x7f93fc0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1d0040 0x55eb6c1eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1ede00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f9411600640 1 -- 172.21.15.63:0/1854506271 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f94080f10c0 con 0x7f9414103060 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c135900 con 0x55eb6c1d0040 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c0dbef0 con 0x55eb6c1eba10 2024-04-25T19:01:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.361+0000 7f941e5d8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c1318c0 con 0x55eb6c1cf720 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.362+0000 7f9419600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 0x7f93fc0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.362+0000 7f9418c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.362+0000 7f9418c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:38640/0 (socket says 172.21.15.63:38640) 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.362+0000 7f9418c00640 1 -- 172.21.15.63:0/1305484821 learned_addr learned my addr 172.21.15.63:0/1305484821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.362+0000 7f9413e00640 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1d0040 0x55eb6c1eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 -- 172.21.15.63:0/1305484821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 msgr2=0x55eb6c1ede00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1ede00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 -- 172.21.15.63:0/1305484821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1d0040 msgr2=0x55eb6c1eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.364 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1d0040 0x55eb6c1eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 -- 172.21.15.63:0/1305484821 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb6be48210 con 0x55eb6c1cf720 2024-04-25T19:01:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9418c00640 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94080c5700 tx=0x7f940803b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e113da67b883d16d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9405c00640 1 -- 172.21.15.63:0/1305484821 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080bcfc0 con 0x55eb6c1cf720 2024-04-25T19:01:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9405c00640 1 -- 172.21.15.63:0/1305484821 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94080dd040 con 0x55eb6c1cf720 2024-04-25T19:01:52.365 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.363+0000 7f9405c00640 1 -- 172.21.15.63:0/1305484821 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940803b4f0 con 0x55eb6c1cf720 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.364+0000 7f941e5d8740 1 -- 172.21.15.63:0/1305484821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 msgr2=0x55eb6c1cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.364+0000 7f941e5d8740 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f94080c5700 tx=0x7f940803b890 comp rx=0 tx=0).stop 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.364+0000 7f9419600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 0x7f93fc0a6300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55eb6c1d12e0 tx=0x7f940c009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 -- 172.21.15.63:0/1305484821 shutdown_connections 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1d0040 0x55eb6c1eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 --2- 172.21.15.63:0/1305484821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c1cfb00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.366 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 -- 172.21.15.63:0/1305484821 >> 172.21.15.63:0/1305484821 conn(0x55eb6c1cbd30 msgr2=0x55eb6c1cc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 -- 172.21.15.63:0/1305484821 shutdown_connections 2024-04-25T19:01:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 -- 172.21.15.63:0/1305484821 wait complete. 2024-04-25T19:01:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.365+0000 7f941e5d8740 1 -- start start 2024-04-25T19:01:52.367 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 Processor -- start 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c1ee400 con 0x55eb6c1cf720 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c0ef310 con 0x55eb6c1eba10 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55eb6c1ee580 con 0x55eb6c1d0040 2024-04-25T19:01:52.368 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f941e5d8740 10 client.0 _ll_register_callbacks cb 0x55eb6c0bdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.366+0000 7f9418c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56836/0 (socket says 172.21.15.63:56836) 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9413e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 -- 172.21.15.63:0/79081004 learned_addr learned my addr 172.21.15.63:0/79081004 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9419600640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 -- 172.21.15.63:0/79081004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 msgr2=0x55eb6c1e8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 -- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 msgr2=0x55eb6c1e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 -- 172.21.15.63:0/79081004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb6bf14620 con 0x55eb6c1cf720 2024-04-25T19:01:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9419600640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:01:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.367+0000 7f9418c00640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f94080bccc0 tx=0x7f9408009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eada2fe3899ba30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.370 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.368+0000 7f93fa000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.368+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94080a1060 con 0x55eb6c1cf720 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.368+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f940803b4f0 con 0x55eb6c1cf720 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.368+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94080dd040 con 0x55eb6c1cf720 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f94080c5c30 con 0x55eb6c1cf720 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f941e5d8740 10 client.16761 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55eb6c1ee400 con 0x55eb6c1cf720 2024-04-25T19:01:52.371 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94080dd040 con 0x55eb6c1cf720 2024-04-25T19:01:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f94080bf070 con 0x55eb6c1cf720 2024-04-25T19:01:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f941e5d8740 10 client.16761 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.369+0000 7f941e5d8740 10 client.16761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f941e5d8740 20 client.16761 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c1fb880 0x55eb6c1fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f941e5d8740 4 client.16761 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb6c1498b0 con 0x55eb6c1fb880 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f9413e00640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c1fb880 0x55eb6c1fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f9413e00640 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c1fb880 0x55eb6c1fdcc0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2adf32af9b752481 server_cookie=433da6f77c44359d in_seq=0 out_seq=0 2024-04-25T19:01:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.370+0000 7f93fa000640 10 client.16761 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:52.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f93fa000640 1 -- 172.21.15.63:0/79081004 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb6c1498b0 con 0x55eb6c1fb880 2024-04-25T19:01:52.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f93fa000640 10 client.16761 handle_command_reply: tid=0 2024-04-25T19:01:52.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 client.16761 shutdown 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 msgr2=0x55eb6c1e8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f94080bccc0 tx=0x7f9408009260 comp rx=0 tx=0).stop 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 shutdown_connections 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c1fb880 0x55eb6c1fdcc0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 --2- 172.21.15.63:0/79081004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e8930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 >> 172.21.15.63:0/79081004 conn(0x55eb6c1cbd30 msgr2=0x55eb6c1d1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.372+0000 7f93fa000640 0 client.16761 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.373+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 shutdown_connections 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.373+0000 7f941e5d8740 1 -- 172.21.15.63:0/79081004 wait complete. 2024-04-25T19:01:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.373+0000 7f941e5d8740 20 client.16761 trim_cache size 0 max 16384 2024-04-25T19:01:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 Processor -- start 2024-04-25T19:01:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 -- start start 2024-04-25T19:01:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94141ae410 con 0x55eb6c1cf720 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94141abc20 con 0x55eb6c1eba10 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.395+0000 7f941e5d8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f941419de40 con 0x55eb6c1d0040 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9418c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9419600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:38656/0 (socket says 172.21.15.63:38656) 2024-04-25T19:01:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 -- 172.21.15.63:0/4189468926 learned_addr learned my addr 172.21.15.63:0/4189468926 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9419600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:56186/0 (socket says 172.21.15.63:56186) 2024-04-25T19:01:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 -- 172.21.15.63:0/4189468926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 msgr2=0x55eb6c1cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 -- 172.21.15.63:0/4189468926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 msgr2=0x55eb6c1e7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 -- 172.21.15.63:0/4189468926 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb6c1c7690 con 0x55eb6c1d0040 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9419600640 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.396+0000 7f9413e00640 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f94141a0240 tx=0x7f9400003f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a239ac5267dc794b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.397+0000 7f9405200640 1 -- 172.21.15.63:0/4189468926 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400014b20 con 0x55eb6c1d0040 2024-04-25T19:01:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.397+0000 7f9405200640 1 -- 172.21.15.63:0/4189468926 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9400014cc0 con 0x55eb6c1d0040 2024-04-25T19:01:52.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.397+0000 7f9405200640 1 -- 172.21.15.63:0/4189468926 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400012720 con 0x55eb6c1d0040 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.398+0000 7f941e5d8740 1 -- 172.21.15.63:0/4189468926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 msgr2=0x55eb6c1e91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.398+0000 7f941e5d8740 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f94141a0240 tx=0x7f9400003f80 comp rx=0 tx=0).stop 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.399+0000 7f941e5d8740 1 -- 172.21.15.63:0/4189468926 shutdown_connections 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.399+0000 7f941e5d8740 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1eba10 0x55eb6c1cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.399+0000 7f941e5d8740 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1d0040 0x55eb6c1e91d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.399+0000 7f941e5d8740 1 --2- 172.21.15.63:0/4189468926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1cf720 0x55eb6c1e7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.399+0000 7f941e5d8740 1 -- 172.21.15.63:0/4189468926 >> 172.21.15.63:0/4189468926 conn(0x55eb6c1cbd30 msgr2=0x55eb6c1ca600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.400+0000 7f941e5d8740 1 -- 172.21.15.63:0/4189468926 shutdown_connections 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.400+0000 7f941e5d8740 1 -- 172.21.15.63:0/4189468926 wait complete. 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.400+0000 7f941e5d8740 1 -- start start 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.400+0000 7f941e5d8740 1 Processor -- start 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 0x55eb6c25a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 0x55eb6c1f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f941419e490 con 0x55eb6c1eba10 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f941419f900 con 0x55eb6c1d0040 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f941419eb60 con 0x55eb6c1cf720 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f941e5d8740 10 client.0 _ll_register_callbacks cb 0x7f94080eebd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f9418c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f9419600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 0x55eb6c1f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f9418c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:38664/0 (socket says 172.21.15.63:38664) 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f9418c00640 1 -- 172.21.15.63:0/215299403 learned_addr learned my addr 172.21.15.63:0/215299403 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.401+0000 7f9413e00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 0x55eb6c25a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 -- 172.21.15.63:0/215299403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 msgr2=0x55eb6c25a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 -- 172.21.15.63:0/215299403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 msgr2=0x55eb6c1f0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 0x55eb6c1f0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 -- 172.21.15.63:0/215299403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb6c1f1130 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9419600640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 0x55eb6c1f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9418c00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f9413e00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 0x55eb6c25a6c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55eb6c1c6d60 tx=0x7f9400014c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f592cdd51bd0465 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.402+0000 7f93fa000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400005460 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(117..117 src has 1..117) ==== 8587+0+0 (secure 0 0 0) 0x7f940001e070 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9400014cc0 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9400014290 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f941e5d8740 10 client.26972 fetch_fsmap learned FSMap version 48 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55eb6c261930 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.403+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9400014cc0 con 0x55eb6c1d0040 2024-04-25T19:01:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f9400004c10 con 0x55eb6c1d0040 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 10 client.26972 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 10 client.26972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 20 client.26972 populate_metadata read hostname 'smithi063' 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c261fd0 0x55eb6c2643f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 4 client.26972 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.404+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eb6c143320 con 0x55eb6c261fd0 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.406+0000 7f9418c00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c261fd0 0x55eb6c2643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.407+0000 7f9418c00640 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c261fd0 0x55eb6c2643f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1261d6c76166de3 server_cookie=973f83e338dbf0f3 in_seq=0 out_seq=0 2024-04-25T19:01:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.407+0000 7f93fa000640 10 client.26972 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.617+0000 7f93fa000640 1 -- 172.21.15.63:0/215299403 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eb6c143320 con 0x55eb6c261fd0 2024-04-25T19:01:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.617+0000 7f93fa000640 10 client.26972 handle_command_reply: tid=0 2024-04-25T19:01:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.617+0000 7f941e5d8740 1 client.26972 shutdown 2024-04-25T19:01:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.617+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 msgr2=0x55eb6c25a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.617+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 0x55eb6c25a6c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x55eb6c1c6d60 tx=0x7f9400014c10 comp rx=0 tx=0).stop 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 shutdown_connections 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55eb6c261fd0 0x55eb6c2643f0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55eb6c1d0040 0x55eb6c25a6c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eb6c1eba10 0x55eb6c1f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 --2- 172.21.15.63:0/215299403 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55eb6c1cf720 0x55eb6c25a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 >> 172.21.15.63:0/215299403 conn(0x55eb6c1cbd30 msgr2=0x55eb6c1e9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f93fa000640 0 client.26972 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 shutdown_connections 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 1 -- 172.21.15.63:0/215299403 wait complete. 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.618+0000 7f941e5d8740 20 client.26972 trim_cache size 0 max 16384 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:01:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:01:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.619+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 msgr2=0x7f93fc0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.619+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 0x7f93fc0a6300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55eb6c1d12e0 tx=0x7f940c009500 comp rx=0 tx=0).stop 2024-04-25T19:01:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.619+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 msgr2=0x7f94141a4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:01:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.619+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9408005ab0 tx=0x7f9408003c80 comp rx=0 tx=0).stop 2024-04-25T19:01:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 shutdown_connections 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f93fc0a3e40 0x7f93fc0a6300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f941410c580 0x7f941410b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9414103980 0x7f94141a5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 --2- 172.21.15.63:0/1854506271 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9414103060 0x7f94141a4af0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 >> 172.21.15.63:0/1854506271 conn(0x7f9414077fd0 msgr2=0x7f94140fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 shutdown_connections 2024-04-25T19:01:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:01:52.620+0000 7f941b600640 1 -- 172.21.15.63:0/1854506271 wait complete. 2024-04-25T19:01:52.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:52 smithi070 ceph-mon[32820]: pgmap v482: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:52.799 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:52 smithi063 ceph-mon[28006]: pgmap v482: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:52 smithi114 ceph-mon[32988]: pgmap v482: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:53.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 46431 24.10 MB/sec warmup 111 sec latency 851.948 ms 2024-04-25T19:01:54.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 47094 24.28 MB/sec warmup 112 sec latency 1852.025 ms 2024-04-25T19:01:54.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:54 smithi070 ceph-mon[32820]: pgmap v483: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:54 smithi114 ceph-mon[32988]: pgmap v483: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:54 smithi063 ceph-mon[28006]: pgmap v483: 97 pgs: 97 active+clean; 521 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 5.0 MiB/s wr, 402 op/s 2024-04-25T19:01:55.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 47570 24.24 MB/sec warmup 113 sec latency 2852.101 ms 2024-04-25T19:01:56.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 48219 24.29 MB/sec warmup 114 sec latency 3852.195 ms 2024-04-25T19:01:56.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:56 smithi070 ceph-mon[32820]: pgmap v484: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-04-25T19:01:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:56 smithi114 ceph-mon[32988]: pgmap v484: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-04-25T19:01:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:56 smithi063 ceph-mon[28006]: pgmap v484: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-04-25T19:01:57.083 INFO:tasks.workunit.client.0.smithi063.stdout: 10 48954 24.47 MB/sec warmup 115 sec latency 4053.823 ms 2024-04-25T19:01:58.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 49566 24.51 MB/sec warmup 116 sec latency 960.249 ms 2024-04-25T19:01:58.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:01:58 smithi070 ceph-mon[32820]: pgmap v485: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.3 MiB/s wr, 485 op/s 2024-04-25T19:01:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:01:58 smithi114 ceph-mon[32988]: pgmap v485: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.3 MiB/s wr, 485 op/s 2024-04-25T19:01:58.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:01:58 smithi063 ceph-mon[28006]: pgmap v485: 97 pgs: 97 active+clean; 525 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.3 MiB/s wr, 485 op/s 2024-04-25T19:01:59.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 50197 24.67 MB/sec warmup 117 sec latency 1960.326 ms 2024-04-25T19:02:00.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 50693 24.66 MB/sec warmup 118 sec latency 2960.393 ms 2024-04-25T19:02:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:00 smithi114 ceph-mon[32988]: pgmap v486: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T19:02:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:00 smithi063 ceph-mon[28006]: pgmap v486: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T19:02:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:00 smithi070 ceph-mon[32820]: pgmap v486: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.5 MiB/s wr, 675 op/s 2024-04-25T19:02:01.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 51352 24.66 MB/sec warmup 119 sec latency 3960.515 ms 2024-04-25T19:02:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:02 smithi114 ceph-mon[32988]: pgmap v487: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:02.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:02 smithi063 ceph-mon[28006]: pgmap v487: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:02 smithi070 ceph-mon[32820]: pgmap v487: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:03.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 52550 40.06 MB/sec execute 1 sec latency 4301.968 ms 2024-04-25T19:02:04.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 53171 37.17 MB/sec execute 2 sec latency 74.181 ms 2024-04-25T19:02:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:04 smithi114 ceph-mon[32988]: pgmap v488: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:04 smithi063 ceph-mon[28006]: pgmap v488: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:04 smithi070 ceph-mon[32820]: pgmap v488: 97 pgs: 97 active+clean; 547 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.3 MiB/s wr, 478 op/s 2024-04-25T19:02:05.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 53473 30.99 MB/sec execute 3 sec latency 337.461 ms 2024-04-25T19:02:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:05 smithi114 ceph-mon[32988]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T19:02:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.842+0000 7f0dbf400640 1 -- 172.21.15.63:0/3763251574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db8107070 msgr2=0x7f0db8107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.842+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3763251574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db8107070 0x7f0db8107450 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0db0009be0 tx=0x7f0db002a6c0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 -- 172.21.15.63:0/3763251574 shutdown_connections 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3763251574 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db81009e0 0x7f0db810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3763251574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db81004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3763251574 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db8107070 0x7f0db8107450 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 -- 172.21.15.63:0/3763251574 >> 172.21.15.63:0/3763251574 conn(0x7f0db8076f50 msgr2=0x7f0db80fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 -- 172.21.15.63:0/3763251574 shutdown_connections 2024-04-25T19:02:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.844+0000 7f0dbf400640 1 -- 172.21.15.63:0/3763251574 wait complete. 2024-04-25T19:02:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.846+0000 7f0dbf400640 1 Processor -- start 2024-04-25T19:02:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.846+0000 7f0dbf400640 1 -- start start 2024-04-25T19:02:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.847+0000 7f0dbf400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db810f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.847+0000 7f0dbf400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 0x7f0db810f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0dbf400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 0x7f0db810fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0dbf400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0db8114450 con 0x7f0db8100020 2024-04-25T19:02:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0dbf400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0db81142d0 con 0x7f0db81009e0 2024-04-25T19:02:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0dbf400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0db81145d0 con 0x7f0db8107070 2024-04-25T19:02:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 0x7f0db810f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 0x7f0db810f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:37236/0 (socket says 172.21.15.63:37236) 2024-04-25T19:02:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.848+0000 7f0db7e00640 1 -- 172.21.15.63:0/3831107982 learned_addr learned my addr 172.21.15.63:0/3831107982 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.849+0000 7f0dbca00640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db810f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.849+0000 7f0dbd400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 0x7f0db810fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbd400640 1 -- 172.21.15.63:0/3831107982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 msgr2=0x7f0db810f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbd400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 0x7f0db810f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbd400640 1 -- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 msgr2=0x7f0db810f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbd400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db810f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbd400640 1 -- 172.21.15.63:0/3831107982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db81aaf00 con 0x7f0db8107070 2024-04-25T19:02:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.850+0000 7f0dbca00640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db810f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:02:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.851+0000 7f0dbd400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 0x7f0db810fef0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0da800e970 tx=0x7f0da800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.851+0000 7f0db5600640 1 -- 172.21.15.63:0/3831107982 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da800ccc0 con 0x7f0db8107070 2024-04-25T19:02:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.851+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db81ab1f0 con 0x7f0db8107070 2024-04-25T19:02:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.851+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db81ab790 con 0x7f0db8107070 2024-04-25T19:02:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.852+0000 7f0db5600640 1 -- 172.21.15.63:0/3831107982 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da8005020 con 0x7f0db8107070 2024-04-25T19:02:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.852+0000 7f0db5600640 1 -- 172.21.15.63:0/3831107982 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8004a00 con 0x7f0db8107070 2024-04-25T19:02:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.857+0000 7f0db5600640 1 -- 172.21.15.63:0/3831107982 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0da8004ba0 con 0x7f0db8107070 2024-04-25T19:02:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.858+0000 7f0dc2230740 1 Processor -- start 2024-04-25T19:02:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.859+0000 7f0dc2230740 1 -- start start 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.860+0000 7f0db5600640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 0x7f0d9c0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.860+0000 7f0db5600640 1 -- 172.21.15.63:0/3831107982 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8631+0+0 (secure 0 0 0) 0x7f0da80d04b0 con 0x7f0db8107070 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a0bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a0c140 0x563458a164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563458971900 con 0x563458a0b820 2024-04-25T19:02:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dbca00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a0bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dbd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dbd400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46864/0 (socket says 172.21.15.63:46864) 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dbd400640 1 -- 172.21.15.63:0/580649593 learned_addr learned my addr 172.21.15.63:0/580649593 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563458917ef0 con 0x563458a0c140 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.861+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56345896d8c0 con 0x563458a16a00 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbd400640 1 -- 172.21.15.63:0/580649593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a0c140 msgr2=0x563458a164c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbd400640 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a0c140 0x563458a164c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbd400640 1 -- 172.21.15.63:0/580649593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 msgr2=0x563458a0bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbd400640 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a0bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbd400640 1 -- 172.21.15.63:0/580649593 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563458684210 con 0x563458a16a00 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0db7e00640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 0x7f0d9c0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.862+0000 7f0dbca00640 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a0bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.863+0000 7f0dbd400640 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0da80987e0 tx=0x7f0da8098750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71b472cb34e8cc8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.863+0000 7f0db7e00640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 0x7f0d9c0a63d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0db8110c30 tx=0x7f0da0009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.863+0000 7f0dadc00640 1 -- 172.21.15.63:0/580649593 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da801c040 con 0x563458a16a00 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.863+0000 7f0dadc00640 1 -- 172.21.15.63:0/580649593 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da80a4ff0 con 0x563458a16a00 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.863+0000 7f0dadc00640 1 -- 172.21.15.63:0/580649593 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80980d0 con 0x563458a16a00 2024-04-25T19:02:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 msgr2=0x563458a18df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0da80987e0 tx=0x7f0da8098750 comp rx=0 tx=0).stop 2024-04-25T19:02:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 shutdown_connections 2024-04-25T19:02:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a0c140 0x563458a164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a16a00 0x563458a18df0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 --2- 172.21.15.63:0/580649593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a0bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.864+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 >> 172.21.15.63:0/580649593 conn(0x563458a07d30 msgr2=0x563458a08110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.865+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 shutdown_connections 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.865+0000 7f0dc2230740 1 -- 172.21.15.63:0/580649593 wait complete. 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.865+0000 7f0dc2230740 1 -- start start 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.865+0000 7f0dc2230740 1 Processor -- start 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.866+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a138e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.866+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 0x563458a13e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.866+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.866+0000 7f0dc2230740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563458a195d0 con 0x563458a0b820 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dc2230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56345892b310 con 0x563458a16a00 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dc2230740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563458a19750 con 0x563458a0c140 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dc2230740 10 client.0 _ll_register_callbacks cb 0x5634588f9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dbd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dbd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:37250/0 (socket says 172.21.15.63:37250) 2024-04-25T19:02:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dbd400640 1 -- 172.21.15.63:0/97870683 learned_addr learned my addr 172.21.15.63:0/97870683 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0db7e00640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 0x563458a13e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.867+0000 7f0dbca00640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a138e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 -- 172.21.15.63:0/97870683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 msgr2=0x563458a13e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 0x563458a13e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 -- 172.21.15.63:0/97870683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 msgr2=0x563458a138e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a138e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 -- 172.21.15.63:0/97870683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563458750620 con 0x563458a16a00 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0db7e00640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 0x563458a13e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dbd400640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0da80b2040 tx=0x7f0da80a07f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2733f772eca8190 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0da6000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:02:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da801c070 con 0x563458a16a00 2024-04-25T19:02:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da80a4ff0 con 0x563458a16a00 2024-04-25T19:02:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.868+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8631+0+0 (secure 0 0 0) 0x7f0da80bb2a0 con 0x563458a16a00 2024-04-25T19:02:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0da809b9e0 con 0x563458a16a00 2024-04-25T19:02:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 10 client.26981 fetch_fsmap learned FSMap version 48 2024-04-25T19:02:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x563458a25750 con 0x563458a16a00 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da80a4ff0 con 0x563458a16a00 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f0da801f030 con 0x563458a16a00 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 10 client.26981 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 10 client.26981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 20 client.26981 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a26aa0 0x563458a28e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 4 client.26981 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.869+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634589858b0 con 0x563458a26aa0 2024-04-25T19:02:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.870+0000 7f0dbca00640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a26aa0 0x563458a28e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.872+0000 7f0dbca00640 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a26aa0 0x563458a28e90 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b63af9cedc6d4c server_cookie=7ddfaec5d919a098 in_seq=0 out_seq=0 2024-04-25T19:02:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.872+0000 7f0da6000640 10 client.26981 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.874+0000 7f0da6000640 1 -- 172.21.15.63:0/97870683 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5634589858b0 con 0x563458a26aa0 2024-04-25T19:02:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.874+0000 7f0da6000640 10 client.26981 handle_command_reply: tid=0 2024-04-25T19:02:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 client.26981 shutdown 2024-04-25T19:02:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 msgr2=0x563458a14360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0da80b2040 tx=0x7f0da80a07f0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 shutdown_connections 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a26aa0 0x563458a28e90 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a14360 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0c140 0x563458a13e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 --2- 172.21.15.63:0/97870683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0b820 0x563458a138e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 >> 172.21.15.63:0/97870683 conn(0x563458a07d30 msgr2=0x563458a0dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.875+0000 7f0da6000640 0 client.26981 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:05.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:05 smithi063 ceph-mon[28006]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T19:02:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.876+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 shutdown_connections 2024-04-25T19:02:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.876+0000 7f0dc2230740 1 -- 172.21.15.63:0/97870683 wait complete. 2024-04-25T19:02:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.876+0000 7f0dc2230740 20 client.26981 trim_cache size 0 max 16384 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.898+0000 7f0dc2230740 1 Processor -- start 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.898+0000 7f0dc2230740 1 -- start start 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.898+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a224c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a2ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dc2230740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d94011220 con 0x563458a0c140 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dc2230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9400ea30 con 0x563458a16a00 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dc2230740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9400c8a0 con 0x563458a0b820 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a224c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a2ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34090/0 (socket says 172.21.15.63:34090) 2024-04-25T19:02:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0db7e00640 1 -- 172.21.15.63:0/2589989071 learned_addr learned my addr 172.21.15.63:0/2589989071 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 -- 172.21.15.63:0/2589989071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 msgr2=0x563458a224c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a224c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 -- 172.21.15.63:0/2589989071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 msgr2=0x563458a2e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0dbd400640 1 -- 172.21.15.63:0/2589989071 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563458a08f60 con 0x563458a16a00 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.899+0000 7f0db7e00640 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.900+0000 7f0dbd400640 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a2ea60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0da809c310 tx=0x7f0da8098750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d6bafd41c1af0c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.900+0000 7f0dad200640 1 -- 172.21.15.63:0/2589989071 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80bb2a0 con 0x563458a16a00 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.900+0000 7f0dad200640 1 -- 172.21.15.63:0/2589989071 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da809b480 con 0x563458a16a00 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.900+0000 7f0dad200640 1 -- 172.21.15.63:0/2589989071 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80a18c0 con 0x563458a16a00 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- 172.21.15.63:0/2589989071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 msgr2=0x563458a2ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a2ea60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0da809c310 tx=0x7f0da8098750 comp rx=0 tx=0).stop 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- 172.21.15.63:0/2589989071 shutdown_connections 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a2ea60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a2e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 --2- 172.21.15.63:0/2589989071 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a224c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- 172.21.15.63:0/2589989071 >> 172.21.15.63:0/2589989071 conn(0x563458a07d30 msgr2=0x563458a0eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- 172.21.15.63:0/2589989071 shutdown_connections 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- 172.21.15.63:0/2589989071 wait complete. 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.901+0000 7f0dc2230740 1 -- start start 2024-04-25T19:02:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 Processor -- start 2024-04-25T19:02:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a1c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9400cef0 con 0x563458a0c140 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9400e360 con 0x563458a16a00 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9400d5c0 con 0x563458a0b820 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.902+0000 7f0dc2230740 10 client.0 _ll_register_callbacks cb 0x7f0da800c7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a1c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:46892/0 (socket says 172.21.15.63:46892) 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0db7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a1c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34100/0 (socket says 172.21.15.63:34100) 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 -- 172.21.15.63:0/1924349970 learned_addr learned my addr 172.21.15.63:0/1924349970 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbd400640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 -- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 msgr2=0x563458a8ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a8ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 -- 172.21.15.63:0/1924349970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 msgr2=0x563458a1c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:05.907 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a1c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.903+0000 7f0dbca00640 1 -- 172.21.15.63:0/1924349970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563458a8b2d0 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0dbca00640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0db002ac80 tx=0x7f0db00307e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d31c05da5d37dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0da6000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0036040 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da809b480 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8631+0+0 (secure 0 0 0) 0x7f0db0003b70 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f0db0003d70 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0dc2230740 10 client.27007 fetch_fsmap learned FSMap version 48 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.904+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x563458a8c5a0 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0da809b480 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 48) ==== 8362+0+0 (secure 0 0 0) 0x7f0db00044d0 con 0x563458a0b820 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 10 client.27007 fetch_fsmap finished waiting for FSMap version 48 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 10 client.27007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:05.908 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 20 client.27007 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a8ccd0 0x563458a8f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 4 client.27007 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56345897f320 con 0x563458a8ccd0 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.905+0000 7f0db7e00640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a8ccd0 0x563458a8f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.906+0000 7f0db7e00640 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a8ccd0 0x563458a8f0f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1aa620ccbb000 server_cookie=bc79c0cbc8dced7d in_seq=0 out_seq=0 2024-04-25T19:02:05.909 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:05.906+0000 7f0da6000640 10 client.27007 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:05.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:05 smithi070 ceph-mon[32820]: osdmap e118: 12 total, 12 up, 12 in 2024-04-25T19:02:06.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 53810 26.43 MB/sec execute 4 sec latency 428.500 ms 2024-04-25T19:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.098+0000 7f0da6000640 1 -- 172.21.15.63:0/1924349970 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56345897f320 con 0x563458a8ccd0 2024-04-25T19:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.098+0000 7f0da6000640 10 client.27007 handle_command_reply: tid=0 2024-04-25T19:02:06.100 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.098+0000 7f0dc2230740 1 client.27007 shutdown 2024-04-25T19:02:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 msgr2=0x563458a1bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:06.102 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0db002ac80 tx=0x7f0db00307e0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 shutdown_connections 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563458a8ccd0 0x563458a8f0f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563458a16a00 0x563458a8ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563458a0c140 0x563458a1c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 --2- 172.21.15.63:0/1924349970 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563458a0b820 0x563458a1bb90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.103 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0da6000640 0 client.27007 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 >> 172.21.15.63:0/1924349970 conn(0x563458a07d30 msgr2=0x563458a13880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.099+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 shutdown_connections 2024-04-25T19:02:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.100+0000 7f0dc2230740 1 -- 172.21.15.63:0/1924349970 wait complete. 2024-04-25T19:02:06.104 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.100+0000 7f0dc2230740 20 client.27007 trim_cache size 0 max 16384 2024-04-25T19:02:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:02:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:02:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:02:06.105 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.100+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 msgr2=0x7f0d9c0a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 0x7f0d9c0a63d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0db8110c30 tx=0x7f0da0009260 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 msgr2=0x7f0db810fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 0x7f0db810fef0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0da800e970 tx=0x7f0da800ee30 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 shutdown_connections 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0d9c0a3f10 0x7f0d9c0a63d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0db81009e0 0x7f0db810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0db8107070 0x7f0db810fef0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 --2- 172.21.15.63:0/3831107982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0db8100020 0x7f0db810f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 >> 172.21.15.63:0/3831107982 conn(0x7f0db8076f50 msgr2=0x7f0db80fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.101+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 shutdown_connections 2024-04-25T19:02:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:06.102+0000 7f0dbf400640 1 -- 172.21.15.63:0/3831107982 wait complete. 2024-04-25T19:02:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:06 smithi114 ceph-mon[32988]: pgmap v490: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:06 smithi063 ceph-mon[28006]: pgmap v490: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:06 smithi070 ceph-mon[32820]: pgmap v490: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:07.084 INFO:tasks.workunit.client.0.smithi063.stdout: 10 54534 26.98 MB/sec execute 5 sec latency 148.371 ms 2024-04-25T19:02:08.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 55276 30.03 MB/sec execute 6 sec latency 48.934 ms 2024-04-25T19:02:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:08 smithi114 ceph-mon[32988]: pgmap v491: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:08.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:08 smithi063 ceph-mon[28006]: pgmap v491: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:08 smithi070 ceph-mon[32820]: pgmap v491: 97 pgs: 97 active+clean; 569 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.8 MiB/s wr, 632 op/s 2024-04-25T19:02:09.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 56068 32.55 MB/sec execute 7 sec latency 78.992 ms 2024-04-25T19:02:10.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 56446 30.87 MB/sec execute 8 sec latency 301.646 ms 2024-04-25T19:02:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:10 smithi114 ceph-mon[32988]: pgmap v492: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:10 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:10 smithi063 ceph-mon[28006]: pgmap v492: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:10.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:10 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:10 smithi070 ceph-mon[32820]: pgmap v492: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:10 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:02:11.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 56966 30.67 MB/sec execute 9 sec latency 117.356 ms 2024-04-25T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:11 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:11 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:11 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:02:11.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:11 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:11.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:11 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:11.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:11 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:02:11.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:11 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:02:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:11 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:02:11.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:11 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:02:12.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 57652 31.25 MB/sec execute 10 sec latency 59.050 ms 2024-04-25T19:02:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:12 smithi114 ceph-mon[32988]: pgmap v493: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:12 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:12 smithi063 ceph-mon[28006]: pgmap v493: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:12 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:12 smithi070 ceph-mon[32820]: pgmap v493: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:12 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:13.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 58356 31.67 MB/sec execute 11 sec latency 41.873 ms 2024-04-25T19:02:14.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 59136 32.72 MB/sec execute 12 sec latency 60.479 ms 2024-04-25T19:02:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:14 smithi114 ceph-mon[32988]: pgmap v494: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:14 smithi114 ceph-mon[32988]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T19:02:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:14 smithi063 ceph-mon[28006]: pgmap v494: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:14 smithi063 ceph-mon[28006]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T19:02:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:14 smithi070 ceph-mon[32820]: pgmap v494: 97 pgs: 97 active+clean; 581 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.9 MiB/s wr, 826 op/s 2024-04-25T19:02:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:14 smithi070 ceph-mon[32820]: osdmap e119: 12 total, 12 up, 12 in 2024-04-25T19:02:15.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 59629 32.15 MB/sec execute 13 sec latency 141.584 ms 2024-04-25T19:02:16.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 60176 32.32 MB/sec execute 14 sec latency 134.695 ms 2024-04-25T19:02:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:16 smithi114 ceph-mon[32988]: pgmap v496: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:16 smithi063 ceph-mon[28006]: pgmap v496: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:16.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:16 smithi070 ceph-mon[32820]: pgmap v496: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:17.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 60756 32.06 MB/sec execute 15 sec latency 138.816 ms 2024-04-25T19:02:18.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 61475 32.15 MB/sec execute 16 sec latency 63.931 ms 2024-04-25T19:02:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:18 smithi114 ceph-mon[32988]: pgmap v497: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:18 smithi063 ceph-mon[28006]: pgmap v497: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:18.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:18 smithi070 ceph-mon[32820]: pgmap v497: 97 pgs: 97 active+clean; 607 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 737 op/s 2024-04-25T19:02:19.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 62280 32.95 MB/sec execute 17 sec latency 42.163 ms 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 -- 172.21.15.63:0/1125346623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 msgr2=0x7fe3141010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 --2- 172.21.15.63:0/1125346623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe3141010c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe304009900 tx=0x7fe30402a410 comp rx=0 tx=0).stop 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 -- 172.21.15.63:0/1125346623 shutdown_connections 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 --2- 172.21.15.63:0/1125346623 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe314115090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 --2- 172.21.15.63:0/1125346623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe314112b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 --2- 172.21.15.63:0/1125346623 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe3141010c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 -- 172.21.15.63:0/1125346623 >> 172.21.15.63:0/1125346623 conn(0x7fe3140fc920 msgr2=0x7fe3140fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.272+0000 7fe31ac00640 1 -- 172.21.15.63:0/1125346623 shutdown_connections 2024-04-25T19:02:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.273+0000 7fe31ac00640 1 -- 172.21.15.63:0/1125346623 wait complete. 2024-04-25T19:02:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.274+0000 7fe31ac00640 1 Processor -- start 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.274+0000 7fe31ac00640 1 -- start start 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe31410dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe31410e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141187e0 con 0x7fe314101690 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe314118660 con 0x7fe314100ce0 2024-04-25T19:02:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe31ac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe314118960 con 0x7fe314106eb0 2024-04-25T19:02:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe313e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe313e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38408/0 (socket says 172.21.15.63:38408) 2024-04-25T19:02:19.277 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe313e00640 1 -- 172.21.15.63:0/4153355051 learned_addr learned my addr 172.21.15.63:0/4153355051 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.275+0000 7fe313400640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe31410dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe318c00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe31410e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313400640 1 -- 172.21.15.63:0/4153355051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 msgr2=0x7fe31410e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313400640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe31410e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313400640 1 -- 172.21.15.63:0/4153355051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 msgr2=0x7fe314114b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313400640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313400640 1 -- 172.21.15.63:0/4153355051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31410e6f0 con 0x7fe314101690 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe318c00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe31410e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:02:19.278 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.276+0000 7fe313e00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe313400640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe31410dbf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe30000e9f0 tx=0x7fe30000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe310c00640 1 -- 172.21.15.63:0/4153355051 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30000ccc0 con 0x7fe314101690 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe310c00640 1 -- 172.21.15.63:0/4153355051 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3000108e0 con 0x7fe314101690 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe310c00640 1 -- 172.21.15.63:0/4153355051 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300018880 con 0x7fe314101690 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31410e9e0 con 0x7fe314101690 2024-04-25T19:02:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.277+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3141129c0 con 0x7fe314101690 2024-04-25T19:02:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.281+0000 7fe31d9b1740 1 Processor -- start 2024-04-25T19:02:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.281+0000 7fe31d9b1740 1 -- start start 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d10aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5605d1007900 con 0x5605d10a1820 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605d0fadef0 con 0x5605d10aca00 2024-04-25T19:02:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5605d10038c0 con 0x5605d10a2140 2024-04-25T19:02:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe313e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe313e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45758/0 (socket says 172.21.15.63:45758) 2024-04-25T19:02:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe313e00640 1 -- 172.21.15.63:0/210778618 learned_addr learned my addr 172.21.15.63:0/210778618 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.282+0000 7fe318c00640 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe313e00640 1 -- 172.21.15.63:0/210778618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 msgr2=0x5605d10aedf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe313e00640 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d10aedf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe313e00640 1 -- 172.21.15.63:0/210778618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 msgr2=0x5605d10a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe313e00640 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe313e00640 1 -- 172.21.15.63:0/210778618 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605d0d1a210 con 0x5605d10a2140 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.283+0000 7fe310c00640 1 -- 172.21.15.63:0/4153355051 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fe300010430 con 0x7fe314101690 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.284+0000 7fe313e00640 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe304006010 tx=0x7fe304004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=311334d803988413 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.284+0000 7fe309200640 1 -- 172.21.15.63:0/210778618 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304040070 con 0x5605d10a2140 2024-04-25T19:02:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.285+0000 7fe309200640 1 -- 172.21.15.63:0/210778618 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3040047a0 con 0x5605d10a2140 2024-04-25T19:02:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.285+0000 7fe309200640 1 -- 172.21.15.63:0/210778618 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304035040 con 0x5605d10a2140 2024-04-25T19:02:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 -- 172.21.15.63:0/210778618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 msgr2=0x5605d10ac4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe304006010 tx=0x7fe304004560 comp rx=0 tx=0).stop 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 -- 172.21.15.63:0/210778618 shutdown_connections 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d10aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10a2140 0x5605d10ac4c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/210778618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe31d9b1740 1 -- 172.21.15.63:0/210778618 >> 172.21.15.63:0/210778618 conn(0x5605d109dd30 msgr2=0x5605d109e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.286+0000 7fe310c00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 0x7fe2f80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.287+0000 7fe310c00640 1 -- 172.21.15.63:0/4153355051 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fe3000d0b70 con 0x7fe314101690 2024-04-25T19:02:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.287+0000 7fe313e00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 0x7fe2f80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.287+0000 7fe31d9b1740 1 -- 172.21.15.63:0/210778618 shutdown_connections 2024-04-25T19:02:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.288+0000 7fe31d9b1740 1 -- 172.21.15.63:0/210778618 wait complete. 2024-04-25T19:02:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.289+0000 7fe31d9b1740 1 -- start start 2024-04-25T19:02:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.290+0000 7fe31d9b1740 1 Processor -- start 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5605d10af5d0 con 0x5605d10a1820 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5605d0fc1310 con 0x5605d10a2140 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5605d10af750 con 0x5605d10aca00 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe31d9b1740 10 client.0 _ll_register_callbacks cb 0x5605d0f8fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe313400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe313400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38422/0 (socket says 172.21.15.63:38422) 2024-04-25T19:02:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe313400640 1 -- 172.21.15.63:0/3742730374 learned_addr learned my addr 172.21.15.63:0/3742730374 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.291+0000 7fe313e00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 0x7fe2f80a6300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3040096c0 tx=0x7fe30402a410 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.292+0000 7fe313400640 1 -- 172.21.15.63:0/3742730374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 msgr2=0x5605d10aa3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:02:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.292+0000 7fe313400640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10aa3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.292+0000 7fe313400640 1 -- 172.21.15.63:0/3742730374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 msgr2=0x5605d10a9950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.292+0000 7fe313e00640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.293+0000 7fe313400640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.293+0000 7fe313400640 1 -- 172.21.15.63:0/3742730374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605d0de6620 con 0x5605d10a2140 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.293+0000 7fe313e00640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.293+0000 7fe2fd600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:02:19.295 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.294+0000 7fe313400640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe3000ce880 tx=0x7fe30009d290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223db9477baba46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.294+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000a5740 con 0x5605d10a2140 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.294+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3040047a0 con 0x5605d10a2140 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.295+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fe30009de00 con 0x5605d10a2140 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.295+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe3000a5740 con 0x5605d10a2140 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.295+0000 7fe31d9b1740 10 client.26996 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:19.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.295+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5605d10bb8a0 con 0x5605d10a2140 2024-04-25T19:02:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.295+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3040047a0 con 0x5605d10a2140 2024-04-25T19:02:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fe300098820 con 0x5605d10a2140 2024-04-25T19:02:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 10 client.26996 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:19.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 10 client.26996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 20 client.26996 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d10bcd10 0x5605d10bf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 4 client.26996 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605d101b8b0 con 0x5605d10bcd10 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.296+0000 7fe313e00640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d10bcd10 0x5605d10bf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.297+0000 7fe313e00640 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d10bcd10 0x5605d10bf100 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfd6b4bef2a3d8a server_cookie=33d9f58085a27121 in_seq=0 out_seq=0 2024-04-25T19:02:19.299 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.297+0000 7fe2fd600640 10 client.26996 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.299+0000 7fe2fd600640 1 -- 172.21.15.63:0/3742730374 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5605d101b8b0 con 0x5605d10bcd10 2024-04-25T19:02:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.299+0000 7fe2fd600640 10 client.26996 handle_command_reply: tid=0 2024-04-25T19:02:19.301 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 client.26996 shutdown 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 msgr2=0x5605d10a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe3000ce880 tx=0x7fe30009d290 comp rx=0 tx=0).stop 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 shutdown_connections 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d10bcd10 0x5605d10bf100 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a2140 0x5605d10a9e90 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe3000ce880 tx=0x7fe30009d290 comp rx=0 tx=0).stop 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe2fd600640 0 client.26996 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3742730374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10a9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 >> 172.21.15.63:0/3742730374 conn(0x5605d109dd30 msgr2=0x5605d10a3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 shutdown_connections 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3742730374 wait complete. 2024-04-25T19:02:19.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.300+0000 7fe31d9b1740 20 client.26996 trim_cache size 0 max 16384 2024-04-25T19:02:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.326+0000 7fe31d9b1740 1 Processor -- start 2024-04-25T19:02:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.326+0000 7fe31d9b1740 1 -- start start 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10a4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10aa020 0x5605d10c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141bf430 con 0x5605d10aa020 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141bcc40 con 0x5605d10a1820 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141baab0 con 0x5605d10aca00 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe313e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe313e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:38436/0 (socket says 172.21.15.63:38436) 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe313e00640 1 -- 172.21.15.63:0/3964619793 learned_addr learned my addr 172.21.15.63:0/3964619793 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe318c00640 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10aa020 0x5605d10c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.327+0000 7fe313400640 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 -- 172.21.15.63:0/3964619793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 msgr2=0x5605d10a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 -- 172.21.15.63:0/3964619793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10aa020 msgr2=0x5605d10c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10aa020 0x5605d10c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 -- 172.21.15.63:0/3964619793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605d10a1f70 con 0x5605d10a1820 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe313e00640 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe304000c00 tx=0x7fe304049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d00bcfdbaecacdd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe2ffe00640 1 -- 172.21.15.63:0/3964619793 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304040040 con 0x5605d10a1820 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe2ffe00640 1 -- 172.21.15.63:0/3964619793 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe304004dd0 con 0x5605d10a1820 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.328+0000 7fe2ffe00640 1 -- 172.21.15.63:0/3964619793 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30405c330 con 0x5605d10a1820 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3964619793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 msgr2=0x5605d10b84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe304000c00 tx=0x7fe304049000 comp rx=0 tx=0).stop 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3964619793 shutdown_connections 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10aa020 0x5605d10c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aca00 0x5605d10a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/3964619793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10a1820 0x5605d10b84c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.329+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3964619793 >> 172.21.15.63:0/3964619793 conn(0x5605d109dd30 msgr2=0x5605d10a9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.330+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3964619793 shutdown_connections 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.330+0000 7fe31d9b1740 1 -- 172.21.15.63:0/3964619793 wait complete. 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.330+0000 7fe31d9b1740 1 -- start start 2024-04-25T19:02:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.330+0000 7fe31d9b1740 1 Processor -- start 2024-04-25T19:02:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 0x5605d10b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d1122270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141bb100 con 0x5605d10a1820 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141bc570 con 0x5605d10aca00 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fe3141bb7d0 con 0x5605d10aa020 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe31d9b1740 10 client.0 _ll_register_callbacks cb 0x7fe300010720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.331+0000 7fe313e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38456/0 (socket says 172.21.15.63:38456) 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 -- 172.21.15.63:0/4097215155 learned_addr learned my addr 172.21.15.63:0/4097215155 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe318c00640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d1122270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313400640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 0x5605d10b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 -- 172.21.15.63:0/4097215155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 msgr2=0x5605d10b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 0x5605d10b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 -- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 msgr2=0x5605d1122270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d1122270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 -- 172.21.15.63:0/4097215155 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605d11227b0 con 0x5605d10a1820 2024-04-25T19:02:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313400640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 0x5605d10b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:02:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe313e00640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe304005880 tx=0x7fe304009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3104dfd1aa2cd97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.332+0000 7fe2fd600640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:02:19.335 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.333+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304040040 con 0x5605d10a1820 2024-04-25T19:02:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.333+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fe304005450 con 0x5605d10a1820 2024-04-25T19:02:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.334+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe304004dd0 con 0x5605d10a1820 2024-04-25T19:02:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.334+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fe3040037b0 con 0x5605d10a1820 2024-04-25T19:02:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.334+0000 7fe31d9b1740 10 client.16797 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:19.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.334+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5605d1123960 con 0x5605d10a1820 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe304004dd0 con 0x5605d10a1820 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fe30404d070 con 0x5605d10a1820 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 10 client.16797 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 20 client.16797 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d1123bd0 0x5605d1125fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 4 client.16797 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5605d1015320 con 0x5605d1123bd0 2024-04-25T19:02:19.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.335+0000 7fe313400640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d1123bd0 0x5605d1125fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.336+0000 7fe313400640 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d1123bd0 0x5605d1125fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f663c45847d483 server_cookie=1b747c880b65f520 in_seq=0 out_seq=0 2024-04-25T19:02:19.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.336+0000 7fe2fd600640 10 client.16797 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.607+0000 7fe2fd600640 1 -- 172.21.15.63:0/4097215155 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5605d1015320 con 0x5605d1123bd0 2024-04-25T19:02:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.607+0000 7fe2fd600640 10 client.16797 handle_command_reply: tid=0 2024-04-25T19:02:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.607+0000 7fe31d9b1740 1 client.16797 shutdown 2024-04-25T19:02:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.608+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 msgr2=0x5605d10b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.608+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe304005880 tx=0x7fe304009780 comp rx=0 tx=0).stop 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 shutdown_connections 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5605d1123bd0 0x5605d1125fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5605d10aca00 0x5605d1122270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5605d10aa020 0x5605d10b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 --2- 172.21.15.63:0/4097215155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5605d10a1820 0x5605d10b27e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 >> 172.21.15.63:0/4097215155 conn(0x5605d109dd30 msgr2=0x5605d109c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe2fd600640 0 client.16797 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 shutdown_connections 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.609+0000 7fe31d9b1740 1 -- 172.21.15.63:0/4097215155 wait complete. 2024-04-25T19:02:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.610+0000 7fe31d9b1740 20 client.16797 trim_cache size 0 max 16384 2024-04-25T19:02:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:02:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:02:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:02:19.612 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:02:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 msgr2=0x7fe2f80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 0x7fe2f80a6300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3040096c0 tx=0x7fe30402a410 comp rx=0 tx=0).stop 2024-04-25T19:02:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 msgr2=0x7fe31410dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe31410dbf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe30000e9f0 tx=0x7fe30000eeb0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.613 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 shutdown_connections 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fe2f80a3e40 0x7fe2f80a6300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fe314106eb0 0x7fe31410e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe314101690 0x7fe31410dbf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 --2- 172.21.15.63:0/4153355051 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe314100ce0 0x7fe314114b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.612+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 >> 172.21.15.63:0/4153355051 conn(0x7fe3140fc920 msgr2=0x7fe3140fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.613+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 shutdown_connections 2024-04-25T19:02:19.614 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:19.613+0000 7fe31ac00640 1 -- 172.21.15.63:0/4153355051 wait complete. 2024-04-25T19:02:20.086 INFO:tasks.workunit.client.0.smithi063.stdout: 10 62635 32.21 MB/sec execute 18 sec latency 259.204 ms 2024-04-25T19:02:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:20 smithi114 ceph-mon[32988]: pgmap v498: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:20.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:20 smithi063 ceph-mon[28006]: pgmap v498: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:20 smithi070 ceph-mon[32820]: pgmap v498: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:21.085 INFO:tasks.workunit.client.0.smithi063.stdout: 10 63256 32.33 MB/sec execute 19 sec latency 178.007 ms 2024-04-25T19:02:22.086 INFO:tasks.workunit.client.0.smithi063.stdout: 10 63990 32.76 MB/sec execute 20 sec latency 52.140 ms 2024-04-25T19:02:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:22 smithi114 ceph-mon[32988]: pgmap v499: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:22.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:22 smithi063 ceph-mon[28006]: pgmap v499: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:22.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:22 smithi070 ceph-mon[32820]: pgmap v499: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:23.086 INFO:tasks.workunit.client.0.smithi063.stdout: 10 64661 32.89 MB/sec execute 21 sec latency 48.457 ms 2024-04-25T19:02:24.086 INFO:tasks.workunit.client.0.smithi063.stdout: 10 65316 33.22 MB/sec execute 22 sec latency 49.093 ms 2024-04-25T19:02:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:24 smithi114 ceph-mon[32988]: pgmap v500: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:24 smithi063 ceph-mon[28006]: pgmap v500: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:24.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:24 smithi070 ceph-mon[32820]: pgmap v500: 97 pgs: 97 active+clean; 636 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-25T19:02:25.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 66010 33.05 MB/sec execute 23 sec latency 70.751 ms 2024-04-25T19:02:25.765 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-04-25T19:02:25.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:02:26.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 66546 32.97 MB/sec execute 24 sec latency 226.677 ms 2024-04-25T19:02:26.351 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:02:26.756 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:26 smithi063 ceph-mon[28006]: pgmap v501: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 819 op/s 2024-04-25T19:02:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:26 smithi114 ceph-mon[32988]: pgmap v501: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 819 op/s 2024-04-25T19:02:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:26 smithi070 ceph-mon[32820]: pgmap v501: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 819 op/s 2024-04-25T19:02:27.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 67300 33.03 MB/sec execute 25 sec latency 51.297 ms 2024-04-25T19:02:28.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 68050 33.66 MB/sec execute 26 sec latency 42.886 ms 2024-04-25T19:02:28.763 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:28 smithi063 ceph-mon[28006]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-25T19:02:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:28 smithi114 ceph-mon[32988]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 -- 172.21.15.63:0/2305123381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b006a750 msgr2=0x7fd3b006ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 --2- 172.21.15.63:0/2305123381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b006a750 0x7fd3b006ab30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd398009990 tx=0x7fd398009ca0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 -- 172.21.15.63:0/2305123381 shutdown_connections 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 --2- 172.21.15.63:0/2305123381 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b010f4f0 0x7fd3b01118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 --2- 172.21.15.63:0/2305123381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b006a750 0x7fd3b006ab30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 --2- 172.21.15.63:0/2305123381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006b070 0x7fd3b010efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 -- 172.21.15.63:0/2305123381 >> 172.21.15.63:0/2305123381 conn(0x7fd3b00788f0 msgr2=0x7fd3b0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 -- 172.21.15.63:0/2305123381 shutdown_connections 2024-04-25T19:02:28.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.851+0000 7fd3b6400700 1 -- 172.21.15.63:0/2305123381 wait complete. 2024-04-25T19:02:28.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.853+0000 7fd3b6400700 1 Processor -- start 2024-04-25T19:02:28.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.853+0000 7fd3b6400700 1 -- start start 2024-04-25T19:02:28.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006a750 0x7fd3b019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b006b070 0x7fd3b019b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0114e50 con 0x7fd3b006a750 2024-04-25T19:02:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0114cd0 con 0x7fd3b010f4f0 2024-04-25T19:02:28.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3b6400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0114fd0 con 0x7fd3b006b070 2024-04-25T19:02:28.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3a6a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b006b070 0x7fd3b019b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:28.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3afe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:28.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3afe00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47612/0 (socket says 172.21.15.63:47612) 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.854+0000 7fd3afe00700 1 -- 172.21.15.63:0/3721574924 learned_addr learned my addr 172.21.15.63:0/3721574924 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3af400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006a750 0x7fd3b019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 -- 172.21.15.63:0/3721574924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b006b070 msgr2=0x7fd3b019b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b006b070 0x7fd3b019b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 -- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006a750 msgr2=0x7fd3b019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006a750 0x7fd3b019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 -- 172.21.15.63:0/3721574924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b01a0370 con 0x7fd3b010f4f0 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3afe00700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800e970 tx=0x7fd3a800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.855+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800cbd0 con 0x7fd3b010f4f0 2024-04-25T19:02:28.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.856+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3a800ee60 con 0x7fd3b010f4f0 2024-04-25T19:02:28.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.856+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a80055d0 con 0x7fd3b010f4f0 2024-04-25T19:02:28.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.856+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b01a0660 con 0x7fd3b010f4f0 2024-04-25T19:02:28.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.856+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b0103b50 con 0x7fd3b010f4f0 2024-04-25T19:02:28.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.858+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b0101f60 con 0x7fd3b010f4f0 2024-04-25T19:02:28.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.860+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd3a8021440 con 0x7fd3b010f4f0 2024-04-25T19:02:28.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.862+0000 7fd3acc00700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 0x7fd3940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:28.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.862+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fd3a80d0ab0 con 0x7fd3b010f4f0 2024-04-25T19:02:28.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.863+0000 7fd3af400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 0x7fd3940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:28.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.865+0000 7fd3af400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 0x7fd3940a6270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd39801a040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:28.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:28.868+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3a809d110 con 0x7fd3b010f4f0 2024-04-25T19:02:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:28 smithi070 ceph-mon[32820]: pgmap v502: 97 pgs: 97 active+clean; 641 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-04-25T19:02:29.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 68676 33.65 MB/sec execute 27 sec latency 49.636 ms 2024-04-25T19:02:29.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.348+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3b019c6c0 con 0x7fd3b010f4f0 2024-04-25T19:02:29.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.349+0000 7fd3acc00700 1 -- 172.21.15.63:0/3721574924 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+11875 (secure 0 0 0) 0x7fd3a809ca10 con 0x7fd3b010f4f0 2024-04-25T19:02:29.351 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:02:29.351 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:02:10.932874+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 msgr2=0x7fd3940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 0x7fd3940a6270 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd398000c00 tx=0x7fd39801a040 comp rx=0 tx=0).stop 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 msgr2=0x7fd3b019fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800e970 tx=0x7fd3a800ec80 comp rx=0 tx=0).stop 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 shutdown_connections 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd3940a3db0 0x7fd3940a6270 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd3b010f4f0 0x7fd3b019fc30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b006a750 0x7fd3b019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:29.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 --2- 172.21.15.63:0/3721574924 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd3b006b070 0x7fd3b019b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:29.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.352+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 >> 172.21.15.63:0/3721574924 conn(0x7fd3b00788f0 msgr2=0x7fd3b01109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:29.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.353+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 shutdown_connections 2024-04-25T19:02:29.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:29.353+0000 7fd3b6400700 1 -- 172.21.15.63:0/3721574924 wait complete. 2024-04-25T19:02:29.355 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 49 2024-04-25T19:02:29.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:29 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3721574924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:02:29.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:29 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3721574924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:02:29.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:29 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3721574924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:02:30.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 69200 33.22 MB/sec execute 28 sec latency 181.641 ms 2024-04-25T19:02:30.487 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 86.5 --await-for 42.0' 2024-04-25T19:02:30.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 86.5 --await-for 42.0 2024-04-25T19:02:30.705 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:30 smithi063 ceph-mon[28006]: pgmap v503: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 955 op/s 2024-04-25T19:02:30.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:30 smithi070 ceph-mon[32820]: pgmap v503: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 955 op/s 2024-04-25T19:02:31.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:30 smithi114 ceph-mon[32988]: pgmap v503: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 955 op/s 2024-04-25T19:02:31.087 INFO:tasks.workunit.client.0.smithi063.stdout: 10 69736 33.10 MB/sec execute 29 sec latency 189.037 ms 2024-04-25T19:02:31.112 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:02:32.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 70537 33.59 MB/sec execute 30 sec latency 46.952 ms 2024-04-25T19:02:32.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:32 smithi063 ceph-mon[28006]: pgmap v504: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:32.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.804+0000 7fdc2fc00640 1 -- 172.21.15.63:0/2142462865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281031e0 msgr2=0x7fdc281035c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.804+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/2142462865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281031e0 0x7fdc281035c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009900 tx=0x7fdc1c02a490 comp rx=0 tx=0).stop 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 -- 172.21.15.63:0/2142462865 shutdown_connections 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/2142462865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc28103b00 0x7fdc2810bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/2142462865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 0x7fdc2810e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/2142462865 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc281031e0 0x7fdc281035c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 -- 172.21.15.63:0/2142462865 >> 172.21.15.63:0/2142462865 conn(0x7fdc28077f60 msgr2=0x7fdc28078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.805+0000 7fdc2fc00640 1 -- 172.21.15.63:0/2142462865 shutdown_connections 2024-04-25T19:02:32.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.806+0000 7fdc2fc00640 1 -- 172.21.15.63:0/2142462865 wait complete. 2024-04-25T19:02:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 Processor -- start 2024-04-25T19:02:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 -- start start 2024-04-25T19:02:32.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc281031e0 0x7fdc281986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 0x7fdc2819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.807+0000 7fdc2fc00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc28112230 con 0x7fdc2810c5b0 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc2fc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281120b0 con 0x7fdc281031e0 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc2fc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281123b0 con 0x7fdc28103b00 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51658/0 (socket says 172.21.15.63:51658) 2024-04-25T19:02:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 -- 172.21.15.63:0/188034602 learned_addr learned my addr 172.21.15.63:0/188034602 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 0x7fdc2819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc2d200640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc281031e0 0x7fdc281986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 -- 172.21.15.63:0/188034602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc281031e0 msgr2=0x7fdc281986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc281031e0 0x7fdc281986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 -- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 msgr2=0x7fdc2819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 0x7fdc2819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.808+0000 7fdc27e00640 1 -- 172.21.15.63:0/188034602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc2819d6e0 con 0x7fdc28103b00 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.809+0000 7fdc27e00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009fa0 tx=0x7fdc1400eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.809+0000 7fdc25600640 1 -- 172.21.15.63:0/188034602 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1400ce10 con 0x7fdc28103b00 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.809+0000 7fdc25600640 1 -- 172.21.15.63:0/188034602 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc140137d0 con 0x7fdc28103b00 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.809+0000 7fdc25600640 1 -- 172.21.15.63:0/188034602 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc14003960 con 0x7fdc28103b00 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.809+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2819d9d0 con 0x7fdc28103b00 2024-04-25T19:02:32.811 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.810+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc28100d60 con 0x7fdc28103b00 2024-04-25T19:02:32.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.816+0000 7fdc32a73740 1 Processor -- start 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 -- start start 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc25600640 1 -- 172.21.15.63:0/188034602 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fdc14022020 con 0x7fdc28103b00 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae5807dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae580884c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae57fe3900 con 0x55ae5807d820 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ae57f89ef0 con 0x55ae58088a00 2024-04-25T19:02:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.817+0000 7fdc32a73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae57fdf8c0 con 0x55ae5807e140 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc27e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc27e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47634/0 (socket says 172.21.15.63:47634) 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc27e00640 1 -- 172.21.15.63:0/2200197572 learned_addr learned my addr 172.21.15.63:0/2200197572 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc2d200640 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae580884c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae5807dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc27e00640 1 -- 172.21.15.63:0/2200197572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 msgr2=0x55ae580884c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.818+0000 7fdc27e00640 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae580884c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc27e00640 1 -- 172.21.15.63:0/2200197572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 msgr2=0x55ae5807dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc27e00640 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae5807dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc27e00640 1 -- 172.21.15.63:0/2200197572 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae57cf6210 con 0x55ae58088a00 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc27e00640 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdc140d1f20 tx=0x7fdc140dab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baaf18230ff9afcc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc25600640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 0x7fdc100a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.819+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 0x7fdc100a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.820+0000 7fdc1a000640 1 -- 172.21.15.63:0/2200197572 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140e4070 con 0x55ae58088a00 2024-04-25T19:02:32.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.820+0000 7fdc25600640 1 -- 172.21.15.63:0/188034602 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fdc140d4790 con 0x7fdc28103b00 2024-04-25T19:02:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.820+0000 7fdc1a000640 1 -- 172.21.15.63:0/2200197572 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1408ca20 con 0x55ae58088a00 2024-04-25T19:02:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.820+0000 7fdc1a000640 1 -- 172.21.15.63:0/2200197572 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc140e4070 con 0x55ae58088a00 2024-04-25T19:02:32.822 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.821+0000 7fdc32a73740 1 -- 172.21.15.63:0/2200197572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 msgr2=0x55ae5808adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.821+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 0x7fdc100a6300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdc2806b1f0 tx=0x7fdc20009560 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.821+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdc140d1f20 tx=0x7fdc140dab40 comp rx=0 tx=0).stop 2024-04-25T19:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 -- 172.21.15.63:0/2200197572 shutdown_connections 2024-04-25T19:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae58088a00 0x55ae5808adf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae580884c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2200197572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae5807dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 -- 172.21.15.63:0/2200197572 >> 172.21.15.63:0/2200197572 conn(0x55ae58079d30 msgr2=0x55ae5807a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 -- 172.21.15.63:0/2200197572 shutdown_connections 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.822+0000 7fdc32a73740 1 -- 172.21.15.63:0/2200197572 wait complete. 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.823+0000 7fdc32a73740 1 -- start start 2024-04-25T19:02:32.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.823+0000 7fdc32a73740 1 Processor -- start 2024-04-25T19:02:32.825 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.824+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.824+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae58085ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae580863e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47646/0 (socket says 172.21.15.63:47646) 2024-04-25T19:02:32.826 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc2d200640 1 -- 172.21.15.63:0/3729451020 learned_addr learned my addr 172.21.15.63:0/3729451020 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae5808b630 con 0x55ae58088a00 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ae57f9d310 con 0x55ae5807d820 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae5808b7b0 con 0x55ae5807e140 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.825+0000 7fdc32a73740 10 client.0 _ll_register_callbacks cb 0x55ae57f6bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc27e00640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae58085ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.827 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae580863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 -- 172.21.15.63:0/3729451020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 msgr2=0x55ae58085ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae58085ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 -- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 msgr2=0x55ae58085960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2dc00640 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae57dc2620 con 0x55ae58088a00 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.826+0000 7fdc2d200640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.827+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae580863e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009fd0 tx=0x7fdc20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3777e737e3fb3a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.828 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.827+0000 7fdc0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20003b70 con 0x55ae58088a00 2024-04-25T19:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fdc20003d10 con 0x55ae58088a00 2024-04-25T19:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1408ca20 con 0x55ae58088a00 2024-04-25T19:02:32.829 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdc2000f7a0 con 0x55ae58088a00 2024-04-25T19:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc32a73740 10 client.16815 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.828+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ae580978f0 con 0x55ae58088a00 2024-04-25T19:02:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1408ca20 con 0x55ae58088a00 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fdc2001f420 con 0x55ae58088a00 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 10 client.16815 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 10 client.16815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 20 client.16815 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae58098e70 0x55ae5809b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 4 client.16815 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.829+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae57ff78b0 con 0x55ae58098e70 2024-04-25T19:02:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.830+0000 7fdc2d200640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae58098e70 0x55ae5809b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.831+0000 7fdc2d200640 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae58098e70 0x55ae5809b260 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1327bb84dea93e8 server_cookie=93bb88b3ab2bc177 in_seq=0 out_seq=0 2024-04-25T19:02:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.831+0000 7fdc0ea00640 10 client.16815 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc0ea00640 1 -- 172.21.15.63:0/3729451020 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae57ff78b0 con 0x55ae58098e70 2024-04-25T19:02:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc0ea00640 10 client.16815 handle_command_reply: tid=0 2024-04-25T19:02:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc32a73740 1 client.16815 shutdown 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 msgr2=0x55ae580863e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae580863e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdc20009fd0 tx=0x7fdc20004a40 comp rx=0 tx=0).stop 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.834+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 shutdown_connections 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae58098e70 0x55ae5809b260 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae580863e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807e140 0x55ae58085ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 --2- 172.21.15.63:0/3729451020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807d820 0x55ae58085960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 >> 172.21.15.63:0/3729451020 conn(0x55ae58079d30 msgr2=0x55ae5807fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc0ea00640 0 client.16815 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 shutdown_connections 2024-04-25T19:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 1 -- 172.21.15.63:0/3729451020 wait complete. 2024-04-25T19:02:32.837 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.835+0000 7fdc32a73740 20 client.16815 trim_cache size 0 max 16384 2024-04-25T19:02:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 Processor -- start 2024-04-25T19:02:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 -- start start 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae58080230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae58088a00 0x55ae580a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281ae500 con 0x55ae5807d820 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281abd10 con 0x55ae5807e140 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.863+0000 7fdc32a73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281a9b80 con 0x55ae58088a00 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50416/0 (socket says 172.21.15.63:50416) 2024-04-25T19:02:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 -- 172.21.15.63:0/2718793250 learned_addr learned my addr 172.21.15.63:0/2718793250 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae58088a00 0x55ae580a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc27e00640 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae58080230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 -- 172.21.15.63:0/2718793250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae58088a00 msgr2=0x55ae580a0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae58088a00 0x55ae580a0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 -- 172.21.15.63:0/2718793250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 msgr2=0x55ae58080230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae58080230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.864+0000 7fdc2d200640 1 -- 172.21.15.63:0/2718793250 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae5807eab0 con 0x55ae5807d820 2024-04-25T19:02:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.865+0000 7fdc2d200640 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000c00 tx=0x7fdc1c005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0f4a0ae54fc80f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.865+0000 7fdc19600640 1 -- 172.21.15.63:0/2718793250 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c035040 con 0x55ae5807d820 2024-04-25T19:02:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.865+0000 7fdc19600640 1 -- 172.21.15.63:0/2718793250 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1c050020 con 0x55ae5807d820 2024-04-25T19:02:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.865+0000 7fdc19600640 1 -- 172.21.15.63:0/2718793250 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c0046c0 con 0x55ae5807d820 2024-04-25T19:02:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.867+0000 7fdc32a73740 1 -- 172.21.15.63:0/2718793250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 msgr2=0x55ae580944f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.867+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000c00 tx=0x7fdc1c005b50 comp rx=0 tx=0).stop 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 -- 172.21.15.63:0/2718793250 shutdown_connections 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae58080230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae58088a00 0x55ae580a0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 --2- 172.21.15.63:0/2718793250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae5807d820 0x55ae580944f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 -- 172.21.15.63:0/2718793250 >> 172.21.15.63:0/2718793250 conn(0x55ae58079d30 msgr2=0x55ae58084230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 -- 172.21.15.63:0/2718793250 shutdown_connections 2024-04-25T19:02:32.869 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 -- 172.21.15.63:0/2718793250 wait complete. 2024-04-25T19:02:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.868+0000 7fdc32a73740 1 -- start start 2024-04-25T19:02:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.869+0000 7fdc32a73740 1 Processor -- start 2024-04-25T19:02:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae5808df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae5808e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281aa1d0 con 0x55ae58088a00 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281ab640 con 0x55ae5807e140 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fdc281aa8a0 con 0x55ae5807d820 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc32a73740 10 client.0 _ll_register_callbacks cb 0x7fdc20012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc2d200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51688/0 (socket says 172.21.15.63:51688) 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc2d200640 1 -- 172.21.15.63:0/642208068 learned_addr learned my addr 172.21.15.63:0/642208068 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc2dc00640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae5808e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.872 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.870+0000 7fdc27e00640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae5808df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 -- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 msgr2=0x55ae5808df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae5808df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 -- 172.21.15.63:0/642208068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 msgr2=0x55ae5808e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae5808e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 -- 172.21.15.63:0/642208068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae580fd5e0 con 0x55ae5807d820 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc27e00640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae5808df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc2d200640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006010 tx=0x7fdc1c0044a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f99d79cbbcec0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.871+0000 7fdc0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:02:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c035040 con 0x55ae5807d820 2024-04-25T19:02:32.874 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1c050020 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fdc1c005440 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fdc1c005640 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc32a73740 10 client.27040 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.872+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55ae580fe890 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc1c050020 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fdc1c004c50 con 0x55ae5807d820 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 10 client.27040 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 10 client.27040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 20 client.27040 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae580fef40 0x55ae58101360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 4 client.27040 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:02:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.873+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae57ff1320 con 0x55ae580fef40 2024-04-25T19:02:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.874+0000 7fdc27e00640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae580fef40 0x55ae58101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.874+0000 7fdc27e00640 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae580fef40 0x55ae58101360 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4717ccc3baaa8a88 server_cookie=84437b40c395f55b in_seq=0 out_seq=0 2024-04-25T19:02:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:32.874+0000 7fdc0ea00640 10 client.27040 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:32.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:32 smithi070 ceph-mon[32820]: pgmap v504: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:32 smithi114 ceph-mon[32988]: pgmap v504: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:33.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71098 33.59 MB/sec execute 31 sec latency 184.994 ms 2024-04-25T19:02:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.205+0000 7fdc0ea00640 1 -- 172.21.15.63:0/642208068 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae57ff1320 con 0x55ae580fef40 2024-04-25T19:02:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.205+0000 7fdc0ea00640 10 client.27040 handle_command_reply: tid=0 2024-04-25T19:02:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.205+0000 7fdc32a73740 1 client.27040 shutdown 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 msgr2=0x55ae5808d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c006010 tx=0x7fdc1c0044a0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 shutdown_connections 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae580fef40 0x55ae58101360 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae5807e140 0x55ae5808df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae58088a00 0x55ae5808e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 --2- 172.21.15.63:0/642208068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae5807d820 0x55ae5808d9d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 >> 172.21.15.63:0/642208068 conn(0x55ae58079d30 msgr2=0x55ae58078e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc0ea00640 0 client.27040 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.206+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 shutdown_connections 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.207+0000 7fdc32a73740 1 -- 172.21.15.63:0/642208068 wait complete. 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.207+0000 7fdc32a73740 20 client.27040 trim_cache size 0 max 16384 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:02:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:02:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.208+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 msgr2=0x7fdc100a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.208+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 0x7fdc100a6300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdc2806b1f0 tx=0x7fdc20009560 comp rx=0 tx=0).stop 2024-04-25T19:02:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.208+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 msgr2=0x7fdc28198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.208+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdc14009fa0 tx=0x7fdc1400eef0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 shutdown_connections 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fdc100a3e40 0x7fdc100a6300 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc2810c5b0 0x7fdc2819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fdc28103b00 0x7fdc28198c30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 --2- 172.21.15.63:0/188034602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdc281031e0 0x7fdc281986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 >> 172.21.15.63:0/188034602 conn(0x7fdc28077f60 msgr2=0x7fdc280fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.209+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 shutdown_connections 2024-04-25T19:02:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:33.210+0000 7fdc2fc00640 1 -- 172.21.15.63:0/188034602 wait complete. 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.670+0000 7f5aab400700 1 -- 172.21.15.63:0/70382456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 msgr2=0x7f5aa4111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.670+0000 7f5aab400700 1 --2- 172.21.15.63:0/70382456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 0x7f5aa4111b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00b0d0 tx=0x7f5a9c00b3e0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.672+0000 7f5aab400700 1 -- 172.21.15.63:0/70382456 shutdown_connections 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.672+0000 7f5aab400700 1 --2- 172.21.15.63:0/70382456 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 0x7f5aa4111b90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.672+0000 7f5aab400700 1 --2- 172.21.15.63:0/70382456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4104ee0 0x7f5aa41052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.672+0000 7f5aab400700 1 --2- 172.21.15.63:0/70382456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5aa4105890 0x7f5aa41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.672+0000 7f5aab400700 1 -- 172.21.15.63:0/70382456 >> 172.21.15.63:0/70382456 conn(0x7f5aa4100770 msgr2=0x7f5aa4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.673+0000 7f5aab400700 1 -- 172.21.15.63:0/70382456 shutdown_connections 2024-04-25T19:02:33.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.673+0000 7f5aab400700 1 -- 172.21.15.63:0/70382456 wait complete. 2024-04-25T19:02:33.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 Processor -- start 2024-04-25T19:02:33.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 -- start start 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5aa4104ee0 0x7f5aa4110490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 0x7f5aa41109d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa4077ed0 con 0x7f5aa4104ee0 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa4077d50 con 0x7f5aa4105890 2024-04-25T19:02:33.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.675+0000 7f5aab400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa4078050 con 0x7f5aa4109fc0 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47676/0 (socket says 172.21.15.63:47676) 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 -- 172.21.15.63:0/1737584465 learned_addr learned my addr 172.21.15.63:0/1737584465 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 -- 172.21.15.63:0/1737584465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 msgr2=0x7f5aa41109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 0x7f5aa41109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 -- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5aa4104ee0 msgr2=0x7f5aa4110490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5aa4104ee0 0x7f5aa4110490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.676+0000 7f5aa3e00700 1 -- 172.21.15.63:0/1737584465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa41b4390 con 0x7f5aa4105890 2024-04-25T19:02:33.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.677+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000ea30 tx=0x7f5a9000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.678+0000 7f5aa1600700 1 -- 172.21.15.63:0/1737584465 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9000cc90 con 0x7f5aa4105890 2024-04-25T19:02:33.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.678+0000 7f5aa1600700 1 -- 172.21.15.63:0/1737584465 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a9000ce30 con 0x7f5aa4105890 2024-04-25T19:02:33.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.678+0000 7f5aa1600700 1 -- 172.21.15.63:0/1737584465 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90010430 con 0x7f5aa4105890 2024-04-25T19:02:33.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.678+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa41b4560 con 0x7f5aa4105890 2024-04-25T19:02:33.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.679+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa41b49f0 con 0x7f5aa4105890 2024-04-25T19:02:33.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.684+0000 7f5aa1600700 1 -- 172.21.15.63:0/1737584465 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5a90003f20 con 0x7f5aa4105890 2024-04-25T19:02:33.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.686+0000 7f5aa1600700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 0x7f5a8c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.686+0000 7f5aa1600700 1 -- 172.21.15.63:0/1737584465 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7f5a900d0a30 con 0x7f5aa4105890 2024-04-25T19:02:33.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.687+0000 7f5ab5d50b80 1 Processor -- start 2024-04-25T19:02:33.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.687+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 0x7f5a8c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.688+0000 7f5ab5d50b80 1 -- start start 2024-04-25T19:02:33.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359c4a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557359b90020 con 0x557359c48180 2024-04-25T19:02:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557359b368c0 con 0x557359c3c9b0 2024-04-25T19:02:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.689+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557359b8bfe0 con 0x557359c3d2d0 2024-04-25T19:02:33.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47692/0 (socket says 172.21.15.63:47692) 2024-04-25T19:02:33.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 -- 172.21.15.63:0/858383894 learned_addr learned my addr 172.21.15.63:0/858383894 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:33.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 0x7f5a8c0a6390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005d90 tx=0x7f5a98005ce0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 -- 172.21.15.63:0/858383894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 msgr2=0x557359c47c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 -- 172.21.15.63:0/858383894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 msgr2=0x557359c4a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.690+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359c4a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.691+0000 7f5aa8a00700 1 -- 172.21.15.63:0/858383894 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557359b2c810 con 0x557359c3c9b0 2024-04-25T19:02:33.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.691+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800ec30 tx=0x7f5a98011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf55ced237dd744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.692+0000 7f5a96000700 1 -- 172.21.15.63:0/858383894 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98020070 con 0x557359c3c9b0 2024-04-25T19:02:33.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.692+0000 7f5a96000700 1 -- 172.21.15.63:0/858383894 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a980108c0 con 0x557359c3c9b0 2024-04-25T19:02:33.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.692+0000 7f5a96000700 1 -- 172.21.15.63:0/858383894 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98003cb0 con 0x557359c3c9b0 2024-04-25T19:02:33.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/858383894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 msgr2=0x557359c3cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800ec30 tx=0x7f5a98011000 comp rx=0 tx=0).stop 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/858383894 shutdown_connections 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359c4a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3c9b0 0x557359c3cd90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/858383894 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.693+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/858383894 >> 172.21.15.63:0/858383894 conn(0x557359b6b730 msgr2=0x557359c38e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.694+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/858383894 shutdown_connections 2024-04-25T19:02:33.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.694+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/858383894 wait complete. 2024-04-25T19:02:33.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.696+0000 7f5ab5d50b80 1 -- start start 2024-04-25T19:02:33.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.696+0000 7f5ab5d50b80 1 Processor -- start 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c44de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 0x557359c457a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557359c4aca0 con 0x557359c45320 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557359b499f0 con 0x557359c48180 2024-04-25T19:02:33.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557359c4ae20 con 0x557359c3d2d0 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.698+0000 7f5ab5d50b80 10 client.0 _ll_register_callbacks cb 0x557359b288b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c44de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51720/0 (socket says 172.21.15.63:51720) 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 -- 172.21.15.63:0/604917379 learned_addr learned my addr 172.21.15.63:0/604917379 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c44de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47696/0 (socket says 172.21.15.63:47696) 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa9400700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 0x557359c457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 -- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 msgr2=0x557359c44de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c44de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.699+0000 7f5aa8a00700 1 -- 172.21.15.63:0/604917379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 msgr2=0x557359c457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.700+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 0x557359c457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.700+0000 7f5aa8a00700 1 -- 172.21.15.63:0/604917379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557359adda60 con 0x557359c3d2d0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.700+0000 7f5aa9400700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 0x557359c457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.701+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e630 tx=0x7f5a98004300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2caf923e4296b10e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.701+0000 7f5a8aa00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.701+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98020070 con 0x557359c3d2d0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.701+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a980108c0 con 0x557359c3d2d0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.701+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7f5a98003b40 con 0x557359c3d2d0 2024-04-25T19:02:33.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.702+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5a98003680 con 0x557359c3d2d0 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.703+0000 7f5ab5d50b80 10 client.27049 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.703+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557359b26c80 con 0x557359c3d2d0 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.703+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a980108c0 con 0x557359c3d2d0 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f5a98005020 con 0x557359c3d2d0 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 10 client.27049 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:33.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 10 client.27049 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:02:33.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 20 client.27049 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:33.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359c581f0 0x557359c5a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 4 client.27049 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:33.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.704+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557359b9d810 con 0x557359c581f0 2024-04-25T19:02:33.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.705+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359c581f0 0x557359c5a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.705+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359c581f0 0x557359c5a670 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ae7882e219e3aba server_cookie=f59194257ae7d15e in_seq=0 out_seq=0 2024-04-25T19:02:33.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.705+0000 7f5a8aa00700 10 client.27049 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:33.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.707+0000 7f5a8aa00700 1 -- 172.21.15.63:0/604917379 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557359b9d810 con 0x557359c581f0 2024-04-25T19:02:33.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5a8aa00700 10 client.27049 handle_command_reply: tid=0 2024-04-25T19:02:33.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 client.27049 shutdown 2024-04-25T19:02:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 msgr2=0x557359c47ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e630 tx=0x7f5a98004300 comp rx=0 tx=0).stop 2024-04-25T19:02:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 shutdown_connections 2024-04-25T19:02:33.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359c581f0 0x557359c5a670 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c3d2d0 0x557359c47ae0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c45320 0x557359c457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/604917379 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c44de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 >> 172.21.15.63:0/604917379 conn(0x557359b6b730 msgr2=0x557359c38a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.708+0000 7f5a8aa00700 0 client.27049 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.709+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 shutdown_connections 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.709+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/604917379 wait complete. 2024-04-25T19:02:33.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.709+0000 7f5ab5d50b80 20 client.27049 trim_cache size 0 max 16384 2024-04-25T19:02:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.736+0000 7f5ab5d50b80 1 Processor -- start 2024-04-25T19:02:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 -- start start 2024-04-25T19:02:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 0x557359c543e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c53bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c500e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41bebd0 con 0x557359c3d2d0 2024-04-25T19:02:33.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41bc3a0 con 0x557359c48180 2024-04-25T19:02:33.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41ba190 con 0x557359c45320 2024-04-25T19:02:33.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.737+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c53bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa3e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c53bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51736/0 (socket says 172.21.15.63:51736) 2024-04-25T19:02:33.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa3e00700 1 -- 172.21.15.63:0/1732816752 learned_addr learned my addr 172.21.15.63:0/1732816752 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 0x557359c543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 -- 172.21.15.63:0/1732816752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 msgr2=0x557359c53bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c53bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 -- 172.21.15.63:0/1732816752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 msgr2=0x557359c543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 0x557359c543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa9400700 1 -- 172.21.15.63:0/1732816752 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557359c5acf0 con 0x557359c48180 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.738+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 0x557359c543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.739+0000 7f5aa9400700 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c500e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00b0d0 tx=0x7f5a9c009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e89af2128a0ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.739+0000 7f5a95600700 1 -- 172.21.15.63:0/1732816752 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c009cf0 con 0x557359c48180 2024-04-25T19:02:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.739+0000 7f5a95600700 1 -- 172.21.15.63:0/1732816752 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a9c003a00 con 0x557359c48180 2024-04-25T19:02:33.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.739+0000 7f5a95600700 1 -- 172.21.15.63:0/1732816752 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c005670 con 0x557359c48180 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1732816752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 msgr2=0x557359c500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c500e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c00b0d0 tx=0x7f5a9c009b80 comp rx=0 tx=0).stop 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1732816752 shutdown_connections 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c48180 0x557359c500e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c3d2d0 0x557359c543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1732816752 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c53bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1732816752 >> 172.21.15.63:0/1732816752 conn(0x557359b6b730 msgr2=0x557359c37f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1732816752 shutdown_connections 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.740+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1732816752 wait complete. 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.741+0000 7f5ab5d50b80 1 -- start start 2024-04-25T19:02:33.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.741+0000 7f5ab5d50b80 1 Processor -- start 2024-04-25T19:02:33.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c4e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359cbd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41ba860 con 0x557359c48180 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41bbcd0 con 0x557359c3d2d0 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa41baf30 con 0x557359c45320 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5ab5d50b80 10 client.0 _ll_register_callbacks cb 0x557359b288b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5aa8a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47710/0 (socket says 172.21.15.63:47710) 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.742+0000 7f5aa8a00700 1 -- 172.21.15.63:0/1859358251 learned_addr learned my addr 172.21.15.63:0/1859358251 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:33.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c4e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 -- 172.21.15.63:0/1859358251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 msgr2=0x557359c4e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c4e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 -- 172.21.15.63:0/1859358251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 msgr2=0x557359cbd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359cbd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 -- 172.21.15.63:0/1859358251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557359cbdd80 con 0x557359c3d2d0 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5aa8a00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5a9801ace0 tx=0x7f5a9801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6b3d141543635 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5a8aa00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:02:33.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.743+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98014040 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7f5a98003b40 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a9c003a00 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5a98047050 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5ab5d50b80 10 client.27044 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557359cbf040 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.744+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a9c003a00 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7f5a98005020 con 0x557359c3d2d0 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 10 client.27044 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 10 client.27044 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 20 client.27044 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:33.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359cbf550 0x557359cc19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 4 client.27044 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.5, "await_for": 42.0}] 2024-04-25T19:02:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 86.5, "await_for": 42.0}) -- 0x557359b978b0 con 0x557359cbf550 2024-04-25T19:02:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.745+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359cbf550 0x557359cc19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.746+0000 7f5aa3e00700 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359cbf550 0x557359cc19a0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50b534020e29765 server_cookie=7c3e0d4eaac10d86 in_seq=0 out_seq=0 2024-04-25T19:02:33.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:33.746+0000 7f5a8aa00700 10 client.27044 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:34.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71417 33.03 MB/sec execute 32 sec latency 327.564 ms 2024-04-25T19:02:34.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.347+0000 7f5a8aa00700 1 -- 172.21.15.63:0/1859358251 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x557359b978b0 con 0x557359cbf550 2024-04-25T19:02:34.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.347+0000 7f5a8aa00700 10 client.27044 handle_command_reply: tid=0 2024-04-25T19:02:34.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.347+0000 7f5ab5d50b80 1 client.27044 shutdown 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.347+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 msgr2=0x557359c4e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.347+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5a9801ace0 tx=0x7f5a9801c420 comp rx=0 tx=0).stop 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 shutdown_connections 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557359cbf550 0x557359cc19a0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557359c48180 0x557359cbd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557359c3d2d0 0x557359c4e2a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5a8aa00700 0 client.27044 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 --2- 172.21.15.63:0/1859358251 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557359c45320 0x557359c4e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 >> 172.21.15.63:0/1859358251 conn(0x557359b6b730 msgr2=0x557359c38820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 shutdown_connections 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 1 -- 172.21.15.63:0/1859358251 wait complete. 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.348+0000 7f5ab5d50b80 20 client.27044 trim_cache size 0 max 16384 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 49, 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 46, 2024-04-25T19:02:34.351 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "aeec7e7c": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 46, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 86.5, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCED", 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:02:34.352 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.349+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 msgr2=0x7f5a8c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.349+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 0x7f5a8c0a6390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5a98005d90 tx=0x7f5a98005ce0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 msgr2=0x7f5aa40744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000ea30 tx=0x7f5a9000edf0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 shutdown_connections 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5a8c0a3ed0 0x7f5a8c0a6390 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5aa4109fc0 0x7f5aa41109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5aa4104ee0 0x7f5aa4110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 --2- 172.21.15.63:0/1737584465 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5aa4105890 0x7f5aa40744c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 >> 172.21.15.63:0/1737584465 conn(0x7f5aa4100770 msgr2=0x7f5aa4102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 shutdown_connections 2024-04-25T19:02:34.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:02:34.350+0000 7f5aab400700 1 -- 172.21.15.63:0/1737584465 wait complete. 2024-04-25T19:02:34.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:34 smithi063 ceph-mon[28006]: pgmap v505: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:34.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:34 smithi070 ceph-mon[32820]: pgmap v505: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:35.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:34 smithi114 ceph-mon[32988]: pgmap v505: 97 pgs: 97 active+clean; 663 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 9.3 MiB/s wr, 650 op/s 2024-04-25T19:02:35.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 32.03 MB/sec execute 33 sec latency 1327.659 ms 2024-04-25T19:02:35.243 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'aeec7e7c', quiesce took 9.5 seconds. Will release after: 26.5 2024-04-25T19:02:36.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 31.09 MB/sec execute 34 sec latency 2327.818 ms 2024-04-25T19:02:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:36 smithi063 ceph-mon[28006]: pgmap v506: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T19:02:36.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:36 smithi070 ceph-mon[32820]: pgmap v506: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T19:02:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:36 smithi114 ceph-mon[32988]: pgmap v506: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 1.01k op/s 2024-04-25T19:02:37.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 30.20 MB/sec execute 35 sec latency 3327.963 ms 2024-04-25T19:02:38.088 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 29.36 MB/sec execute 36 sec latency 4328.127 ms 2024-04-25T19:02:38.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:38 smithi063 ceph-mon[28006]: pgmap v507: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 634 op/s 2024-04-25T19:02:38.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:38 smithi070 ceph-mon[32820]: pgmap v507: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 634 op/s 2024-04-25T19:02:39.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:38 smithi114 ceph-mon[32988]: pgmap v507: 97 pgs: 97 active+clean; 668 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 8.7 MiB/s wr, 634 op/s 2024-04-25T19:02:39.089 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 28.57 MB/sec execute 37 sec latency 5328.228 ms 2024-04-25T19:02:40.089 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 27.82 MB/sec execute 38 sec latency 6328.380 ms 2024-04-25T19:02:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:40 smithi063 ceph-mon[28006]: pgmap v508: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 724 op/s 2024-04-25T19:02:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:40 smithi070 ceph-mon[32820]: pgmap v508: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 724 op/s 2024-04-25T19:02:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:40 smithi114 ceph-mon[32988]: pgmap v508: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 11 MiB/s wr, 724 op/s 2024-04-25T19:02:41.089 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 27.11 MB/sec execute 39 sec latency 7328.674 ms 2024-04-25T19:02:42.089 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 26.43 MB/sec execute 40 sec latency 8328.872 ms 2024-04-25T19:02:42.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:42 smithi114 ceph-mon[32988]: pgmap v509: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:42 smithi063 ceph-mon[28006]: pgmap v509: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:42 smithi070 ceph-mon[32820]: pgmap v509: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:43.089 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 25.78 MB/sec execute 41 sec latency 9329.013 ms 2024-04-25T19:02:44.090 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 25.17 MB/sec execute 42 sec latency 10329.146 ms 2024-04-25T19:02:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:44 smithi063 ceph-mon[28006]: pgmap v510: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:44.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:44 smithi070 ceph-mon[32820]: pgmap v510: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:44 smithi114 ceph-mon[32988]: pgmap v510: 97 pgs: 97 active+clean; 669 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.4 MiB/s wr, 447 op/s 2024-04-25T19:02:45.090 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 24.58 MB/sec execute 43 sec latency 11329.303 ms 2024-04-25T19:02:46.090 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 24.03 MB/sec execute 44 sec latency 12329.478 ms 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 -- 172.21.15.63:0/150922424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec104d50 msgr2=0x7fb1ec108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 --2- 172.21.15.63:0/150922424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec104d50 0x7fb1ec108d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009860 tx=0x7fb1d402a340 comp rx=0 tx=0).stop 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 -- 172.21.15.63:0/150922424 shutdown_connections 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 --2- 172.21.15.63:0/150922424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 0x7fb1ec111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 --2- 172.21.15.63:0/150922424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec104d50 0x7fb1ec108d90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.397 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 --2- 172.21.15.63:0/150922424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 -- 172.21.15.63:0/150922424 >> 172.21.15.63:0/150922424 conn(0x7fb1ec0ffc40 msgr2=0x7fb1ec102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 -- 172.21.15.63:0/150922424 shutdown_connections 2024-04-25T19:02:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.394+0000 7fb1f3000640 1 -- 172.21.15.63:0/150922424 wait complete. 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.395+0000 7fb1f3000640 1 Processor -- start 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.395+0000 7fb1f3000640 1 -- start start 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 0x7fb1ec1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 0x7fb1ec19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec114390 con 0x7fb1ec109470 2024-04-25T19:02:46.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec114210 con 0x7fb1ec1043a0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f3000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec114510 con 0x7fb1ec19a710 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40932/0 (socket says 172.21.15.63:40932) 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1ebe00640 1 -- 172.21.15.63:0/4273720581 learned_addr learned my addr 172.21.15.63:0/4273720581 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 0x7fb1ec1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.396+0000 7fb1f1000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 0x7fb1ec19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 -- 172.21.15.63:0/4273720581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 msgr2=0x7fb1ec19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 0x7fb1ec19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 -- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 msgr2=0x7fb1ec1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 0x7fb1ec1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 -- 172.21.15.63:0/4273720581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec19b310 con 0x7fb1ec1043a0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1f1000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 0x7fb1ec19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9f0 tx=0x7fb1dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1e9600640 1 -- 172.21.15.63:0/4273720581 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00cd80 con 0x7fb1ec1043a0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec1a73f0 con 0x7fb1ec1043a0 2024-04-25T19:02:46.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1e9600640 1 -- 172.21.15.63:0/4273720581 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1dc0108e0 con 0x7fb1ec1043a0 2024-04-25T19:02:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1e9600640 1 -- 172.21.15.63:0/4273720581 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc018980 con 0x7fb1ec1043a0 2024-04-25T19:02:46.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.397+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec1a77a0 con 0x7fb1ec1043a0 2024-04-25T19:02:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 Processor -- start 2024-04-25T19:02:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1e9600640 1 -- 172.21.15.63:0/4273720581 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb1dc010430 con 0x7fb1ec1043a0 2024-04-25T19:02:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 -- start start 2024-04-25T19:02:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fed3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.403 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 0x557d7fede4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.404 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.404 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557d7fe39900 con 0x557d7fedea00 2024-04-25T19:02:46.404 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557d7fddfef0 con 0x557d7fed4140 2024-04-25T19:02:46.404 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557d7fe358c0 con 0x557d7fed3820 2024-04-25T19:02:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 0x557d7fede4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45130/0 (socket says 172.21.15.63:45130) 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.401+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fed3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 -- 172.21.15.63:0/3821895662 learned_addr learned my addr 172.21.15.63:0/3821895662 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 -- 172.21.15.63:0/3821895662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 msgr2=0x557d7fed3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fed3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 -- 172.21.15.63:0/3821895662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 msgr2=0x557d7fede4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 0x557d7fede4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 -- 172.21.15.63:0/3821895662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7fb4c210 con 0x557d7fedea00 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1f1000640 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fed3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 0x557d7fede4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06b6c0 tx=0x7fb1d4005b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e998650c81c65cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.402+0000 7fb1e9600640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 0x7fb1d00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.403+0000 7fb1e1600640 1 -- 172.21.15.63:0/3821895662 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4037040 con 0x557d7fedea00 2024-04-25T19:02:46.406 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.403+0000 7fb1e9600640 1 -- 172.21.15.63:0/4273720581 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fb1dc0d0b00 con 0x7fb1ec1043a0 2024-04-25T19:02:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.403+0000 7fb1e1600640 1 -- 172.21.15.63:0/3821895662 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1d4030450 con 0x557d7fedea00 2024-04-25T19:02:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.403+0000 7fb1e1600640 1 -- 172.21.15.63:0/3821895662 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4003970 con 0x557d7fedea00 2024-04-25T19:02:46.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.403+0000 7fb1f1000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 0x7fb1d00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3821895662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 msgr2=0x557d7fee0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec06b6c0 tx=0x7fb1d4005b10 comp rx=0 tx=0).stop 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f1000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 0x7fb1d00a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005fd0 tx=0x7fb1e4010040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3821895662 shutdown_connections 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fed4140 0x557d7fede4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fedea00 0x557d7fee0df0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3821895662 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fed3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3821895662 >> 172.21.15.63:0/3821895662 conn(0x557d7fecfd30 msgr2=0x557d7fed0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3821895662 shutdown_connections 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3821895662 wait complete. 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 -- start start 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.404+0000 7fb1f5e21740 1 Processor -- start 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fedbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fedc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557d7fee15d0 con 0x557d7fed4140 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557d7fdf3310 con 0x557d7fedea00 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x557d7fee1750 con 0x557d7fed3820 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f5e21740 10 client.0 _ll_register_callbacks cb 0x557d7fdc1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fedbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58416/0 (socket says 172.21.15.63:58416) 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1ebe00640 1 -- 172.21.15.63:0/30163441 learned_addr learned my addr 172.21.15.63:0/30163441 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.405+0000 7fb1f1000640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fedc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 -- 172.21.15.63:0/30163441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 msgr2=0x557d7fedb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 -- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 msgr2=0x557d7fedc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fedc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 -- 172.21.15.63:0/30163441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7fc18620 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1f1000640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fedc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fedbea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4035040 tx=0x7fb1d402aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb2228ee7308ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1da000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4037030 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fb1d4005230 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.406+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d4030450 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb1d4056050 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 10 client.16839 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557d7feed7f0 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1d4030450 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fb1d403d480 con 0x557d7fed4140 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 10 client.16839 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:46.413 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 20 client.16839 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7feeeb40 0x557d7fef0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 4 client.16839 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.407+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d7fe4d8b0 con 0x557d7feeeb40 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.408+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7feeeb40 0x557d7fef0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.408+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7feeeb40 0x557d7fef0f30 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df65ad821b3164dd server_cookie=74af5e8cf8153bb1 in_seq=0 out_seq=0 2024-04-25T19:02:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.408+0000 7fb1da000640 10 client.16839 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1da000640 1 -- 172.21.15.63:0/30163441 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d7fe4d8b0 con 0x557d7feeeb40 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1da000640 10 client.16839 handle_command_reply: tid=0 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 client.16839 shutdown 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 msgr2=0x557d7fedbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fedbea0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4035040 tx=0x7fb1d402aab0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 shutdown_connections 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7feeeb40 0x557d7fef0f30 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fedc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fedbea0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/30163441 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fedb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1da000640 0 client.16839 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 >> 172.21.15.63:0/30163441 conn(0x557d7fecfd30 msgr2=0x557d7fed5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.409+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 shutdown_connections 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.410+0000 7fb1f5e21740 1 -- 172.21.15.63:0/30163441 wait complete. 2024-04-25T19:02:46.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.410+0000 7fb1f5e21740 20 client.16839 trim_cache size 0 max 16384 2024-04-25T19:02:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.426+0000 7fb1f5e21740 1 Processor -- start 2024-04-25T19:02:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 -- start start 2024-04-25T19:02:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7feea490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fef66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fef6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1b2710 con 0x557d7fed4140 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1aff20 con 0x557d7fedea00 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1add90 con 0x557d7fed3820 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fef66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fef6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7feea490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fef66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45138/0 (socket says 172.21.15.63:45138) 2024-04-25T19:02:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.427+0000 7fb1e3e00640 1 -- 172.21.15.63:0/724796830 learned_addr learned my addr 172.21.15.63:0/724796830 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 -- 172.21.15.63:0/724796830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 msgr2=0x557d7fef6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fef6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 -- 172.21.15.63:0/724796830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 msgr2=0x557d7fef66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fef66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 -- 172.21.15.63:0/724796830 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7fed3f70 con 0x557d7fed3820 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7feea490 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9c0 tx=0x7fb1dc0d3010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f219a13d18ee9d1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1e0c00640 1 -- 172.21.15.63:0/724796830 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc09c470 con 0x557d7fed3820 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1e0c00640 1 -- 172.21.15.63:0/724796830 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1dc010720 con 0x557d7fed3820 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.428+0000 7fb1e0c00640 1 -- 172.21.15.63:0/724796830 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc010a90 con 0x557d7fed3820 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- 172.21.15.63:0/724796830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 msgr2=0x557d7feea490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7feea490 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9c0 tx=0x7fb1dc0d3010 comp rx=0 tx=0).stop 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- 172.21.15.63:0/724796830 shutdown_connections 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fef6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fef66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/724796830 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7feea490 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- 172.21.15.63:0/724796830 >> 172.21.15.63:0/724796830 conn(0x557d7fecfd30 msgr2=0x557d7fed0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- 172.21.15.63:0/724796830 shutdown_connections 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- 172.21.15.63:0/724796830 wait complete. 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.429+0000 7fb1f5e21740 1 -- start start 2024-04-25T19:02:46.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 Processor -- start 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fee6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fee3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fee4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1ae3e0 con 0x557d7fed4140 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1af850 con 0x557d7fedea00 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec1aeab0 con 0x557d7fed3820 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f5e21740 10 client.0 _ll_register_callbacks cb 0x7fb1dc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1e3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fee3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fee4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.430+0000 7fb1ebe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fee6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1f1000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fee4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:40988/0 (socket says 172.21.15.63:40988) 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1f1000640 1 -- 172.21.15.63:0/3703242763 learned_addr learned my addr 172.21.15.63:0/3703242763 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:46.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 -- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 msgr2=0x557d7fee4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fee4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 -- 172.21.15.63:0/3703242763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 msgr2=0x557d7fee3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fee3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 -- 172.21.15.63:0/3703242763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7fee49d0 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1ebe00640 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fee6a20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0994e0 tx=0x7fb1dc01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ea2f3b13dd96c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1da000640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc09c470 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fb1dc010a90 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.431+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557d7fe47320 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb1dc09cb10 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 10 client.27073 fetch_fsmap learned FSMap version 49 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x557d7ff54c20 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x557d7fe47320 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 49) ==== 8358+0+0 (secure 0 0 0) 0x7fb1dc018b20 con 0x557d7fed3820 2024-04-25T19:02:46.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 10 client.27073 fetch_fsmap finished waiting for FSMap version 49 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 10 client.27073 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 20 client.27073 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7ff55250 0x557d7ff57670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 4 client.27073 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d7fe41190 con 0x557d7ff55250 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.432+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7ff55250 0x557d7ff57670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.433+0000 7fb1e3e00640 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7ff55250 0x557d7ff57670 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da49161d7f74ed6c server_cookie=fcef08d2eea05b93 in_seq=0 out_seq=0 2024-04-25T19:02:46.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.433+0000 7fb1da000640 10 client.27073 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1da000640 1 -- 172.21.15.63:0/3703242763 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d7fe41190 con 0x557d7ff55250 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1da000640 10 client.27073 handle_command_reply: tid=0 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 client.27073 shutdown 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 msgr2=0x557d7fee6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fee6a20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0994e0 tx=0x7fb1dc01f000 comp rx=0 tx=0).stop 2024-04-25T19:02:46.514 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 shutdown_connections 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557d7ff55250 0x557d7ff57670 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557d7fedea00 0x557d7fee4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557d7fed4140 0x557d7fee3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 --2- 172.21.15.63:0/3703242763 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x557d7fed3820 0x557d7fee6a20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1da000640 0 client.27073 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.511+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 >> 172.21.15.63:0/3703242763 conn(0x557d7fecfd30 msgr2=0x557d7fedb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 shutdown_connections 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f5e21740 1 -- 172.21.15.63:0/3703242763 wait complete. 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f5e21740 20 client.27073 trim_cache size 0 max 16384 2024-04-25T19:02:46.515 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:02:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:02:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:02:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:02:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 msgr2=0x7fb1d00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 0x7fb1d00a6280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4005fd0 tx=0x7fb1e4010040 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 msgr2=0x7fb1ec1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.512+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00e9f0 tx=0x7fb1dc00eeb0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 shutdown_connections 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb1d00a3dc0 0x7fb1d00a6280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb1ec19a710 0x7fb1ec19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1ec109470 0x7fb1ec1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 --2- 172.21.15.63:0/4273720581 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1ec1043a0 0x7fb1ec1a0650 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 >> 172.21.15.63:0/4273720581 conn(0x7fb1ec0ffc40 msgr2=0x7fb1ec100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 shutdown_connections 2024-04-25T19:02:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:46.513+0000 7fb1f3000640 1 -- 172.21.15.63:0/4273720581 wait complete. 2024-04-25T19:02:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:46 smithi063 ceph-mon[28006]: pgmap v511: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 539 op/s 2024-04-25T19:02:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:46 smithi070 ceph-mon[32820]: pgmap v511: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 539 op/s 2024-04-25T19:02:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:46 smithi114 ceph-mon[32988]: pgmap v511: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 539 op/s 2024-04-25T19:02:47.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 23.49 MB/sec execute 45 sec latency 13329.643 ms 2024-04-25T19:02:48.090 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 22.98 MB/sec execute 46 sec latency 14329.836 ms 2024-04-25T19:02:48.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:48 smithi063 ceph-mon[28006]: pgmap v512: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 5.0 MiB/s wr, 182 op/s 2024-04-25T19:02:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:48 smithi070 ceph-mon[32820]: pgmap v512: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 5.0 MiB/s wr, 182 op/s 2024-04-25T19:02:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:48 smithi114 ceph-mon[32988]: pgmap v512: 97 pgs: 97 active+clean; 667 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 5.0 MiB/s wr, 182 op/s 2024-04-25T19:02:49.090 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 22.49 MB/sec execute 47 sec latency 15330.001 ms 2024-04-25T19:02:50.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 22.02 MB/sec execute 48 sec latency 16330.160 ms 2024-04-25T19:02:50.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:50 smithi070 ceph-mon[32820]: pgmap v513: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 7.5 MiB/s wr, 284 op/s 2024-04-25T19:02:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:50 smithi114 ceph-mon[32988]: pgmap v513: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 7.5 MiB/s wr, 284 op/s 2024-04-25T19:02:51.071 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:50 smithi063 ceph-mon[28006]: pgmap v513: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 7.5 MiB/s wr, 284 op/s 2024-04-25T19:02:51.093 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 21.57 MB/sec execute 49 sec latency 17330.292 ms 2024-04-25T19:02:52.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 21.14 MB/sec execute 50 sec latency 18330.372 ms 2024-04-25T19:02:52.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:52 smithi063 ceph-mon[28006]: pgmap v514: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-04-25T19:02:52.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:52.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:52 smithi070 ceph-mon[32820]: pgmap v514: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-04-25T19:02:52.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:52 smithi114 ceph-mon[32988]: pgmap v514: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 193 op/s 2024-04-25T19:02:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:02:53.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 20.73 MB/sec execute 51 sec latency 19330.504 ms 2024-04-25T19:02:54.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 20.33 MB/sec execute 52 sec latency 20330.602 ms 2024-04-25T19:02:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:54 smithi070 ceph-mon[32820]: pgmap v515: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 194 op/s 2024-04-25T19:02:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:54 smithi114 ceph-mon[32988]: pgmap v515: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 194 op/s 2024-04-25T19:02:55.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 19.95 MB/sec execute 53 sec latency 21330.718 ms 2024-04-25T19:02:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:54 smithi063 ceph-mon[28006]: pgmap v515: 97 pgs: 97 active+clean; 681 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 5.2 MiB/s wr, 194 op/s 2024-04-25T19:02:56.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 19.58 MB/sec execute 54 sec latency 22330.836 ms 2024-04-25T19:02:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:56 smithi070 ceph-mon[32820]: pgmap v516: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 7.7 MiB/s wr, 272 op/s 2024-04-25T19:02:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:56 smithi114 ceph-mon[32988]: pgmap v516: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 7.7 MiB/s wr, 272 op/s 2024-04-25T19:02:57.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 19.22 MB/sec execute 55 sec latency 23330.955 ms 2024-04-25T19:02:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:56 smithi063 ceph-mon[28006]: pgmap v516: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 7.7 MiB/s wr, 272 op/s 2024-04-25T19:02:57.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:02:57 smithi070 ceph-mon[32820]: pgmap v517: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 4.9 MiB/s wr, 180 op/s 2024-04-25T19:02:58.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:02:57 smithi114 ceph-mon[32988]: pgmap v517: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 4.9 MiB/s wr, 180 op/s 2024-04-25T19:02:58.091 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 18.88 MB/sec execute 56 sec latency 24331.086 ms 2024-04-25T19:02:58.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:02:57 smithi063 ceph-mon[28006]: pgmap v517: 97 pgs: 97 active+clean; 687 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 4.9 MiB/s wr, 180 op/s 2024-04-25T19:02:59.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 18.55 MB/sec execute 57 sec latency 25331.187 ms 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 -- 172.21.15.63:0/2459324471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa264100b20 msgr2=0x7fa264100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 --2- 172.21.15.63:0/2459324471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa264100b20 0x7fa264100f00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa254009be0 tx=0x7fa25402a6c0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 -- 172.21.15.63:0/2459324471 shutdown_connections 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 --2- 172.21.15.63:0/2459324471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2641014d0 0x7fa26410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 --2- 172.21.15.63:0/2459324471 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 0x7fa264110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 --2- 172.21.15.63:0/2459324471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa264100b20 0x7fa264100f00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 -- 172.21.15.63:0/2459324471 >> 172.21.15.63:0/2459324471 conn(0x7fa2640fc770 msgr2=0x7fa2640feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 -- 172.21.15.63:0/2459324471 shutdown_connections 2024-04-25T19:02:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.659+0000 7fa26a400640 1 -- 172.21.15.63:0/2459324471 wait complete. 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.660+0000 7fa26a400640 1 Processor -- start 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.660+0000 7fa26a400640 1 -- start start 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2641014d0 0x7fa2641a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 0x7fa26419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa264114200 con 0x7fa2641014d0 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa264114080 con 0x7fa264100b20 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa26a400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa264114380 con 0x7fa26410e6d0 2024-04-25T19:02:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:48194/0 (socket says 172.21.15.63:48194) 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 -- 172.21.15.63:0/253856309 learned_addr learned my addr 172.21.15.63:0/253856309 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa263e00640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2641014d0 0x7fa2641a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269a00640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 0x7fa26419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 -- 172.21.15.63:0/253856309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 msgr2=0x7fa26419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 0x7fa26419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 -- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2641014d0 msgr2=0x7fa2641a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2641014d0 0x7fa2641a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.661+0000 7fa269000640 1 -- 172.21.15.63:0/253856309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26419ade0 con 0x7fa264100b20 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa269000640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa25402acb0 tx=0x7fa254003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa261600640 1 -- 172.21.15.63:0/253856309 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254041020 con 0x7fa264100b20 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26419b070 con 0x7fa264100b20 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa261600640 1 -- 172.21.15.63:0/253856309 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2540318f0 con 0x7fa264100b20 2024-04-25T19:02:59.666 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa261600640 1 -- 172.21.15.63:0/253856309 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254036040 con 0x7fa264100b20 2024-04-25T19:02:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.662+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2641a7320 con 0x7fa264100b20 2024-04-25T19:02:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.665+0000 7fa26e97d740 1 Processor -- start 2024-04-25T19:02:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa261600640 1 -- 172.21.15.63:0/253856309 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa254044020 con 0x7fa264100b20 2024-04-25T19:02:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 -- start start 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 0x5570440b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440bbb90 0x5570440bdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557044016900 con 0x5570440b09b0 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557043fbcef0 con 0x5570440b12d0 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa26e97d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5570440128c0 con 0x5570440bbb90 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 0x5570440b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.666+0000 7fa269000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa263e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440bbb90 0x5570440bdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 0x5570440b0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56268/0 (socket says 172.21.15.63:56268) 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269a00640 1 -- 172.21.15.63:0/3879576383 learned_addr learned my addr 172.21.15.63:0/3879576383 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 -- 172.21.15.63:0/3879576383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440bbb90 msgr2=0x5570440bdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440bbb90 0x5570440bdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 -- 172.21.15.63:0/3879576383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 msgr2=0x5570440b0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 0x5570440b0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 -- 172.21.15.63:0/3879576383 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557043d29210 con 0x5570440b12d0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa269000640 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440bb650 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ee9c0 tx=0x7fa2540f95a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf44df70286acff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.667+0000 7fa261600640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 0x7fa2480a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa25a000640 1 -- 172.21.15.63:0/3879576383 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540b94b0 con 0x5570440b12d0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa25a000640 1 -- 172.21.15.63:0/3879576383 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2540bd910 con 0x5570440b12d0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa25a000640 1 -- 172.21.15.63:0/3879576383 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540bc990 con 0x5570440b12d0 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa261600640 1 -- 172.21.15.63:0/253856309 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fa2540f1310 con 0x7fa264100b20 2024-04-25T19:02:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa269a00640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 0x7fa2480a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa26e97d740 1 -- 172.21.15.63:0/3879576383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 msgr2=0x5570440bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.668+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440bb650 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ee9c0 tx=0x7fa2540f95a0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 -- 172.21.15.63:0/3879576383 shutdown_connections 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440bb650 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440bbb90 0x5570440bdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3879576383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440b09b0 0x5570440b0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 -- 172.21.15.63:0/3879576383 >> 172.21.15.63:0/3879576383 conn(0x5570440acec0 msgr2=0x5570440ad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa269a00640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 0x7fa2480a61d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009dc0 tx=0x7fa25c009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 -- 172.21.15.63:0/3879576383 shutdown_connections 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 -- 172.21.15.63:0/3879576383 wait complete. 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 -- start start 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.669+0000 7fa26e97d740 1 Processor -- start 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5570440be580 con 0x5570440bbb90 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x557043fd0310 con 0x5570440b09b0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5570440be700 con 0x5570440b12d0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa26e97d740 10 client.0 _ll_register_callbacks cb 0x557043f9ecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56276/0 (socket says 172.21.15.63:56276) 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269a00640 1 -- 172.21.15.63:0/3843335954 learned_addr learned my addr 172.21.15.63:0/3843335954 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa263e00640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269000640 1 -- 172.21.15.63:0/3843335954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 msgr2=0x5570440b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269000640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440b8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.670+0000 7fa269000640 1 -- 172.21.15.63:0/3843335954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 msgr2=0x5570440b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa269000640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa269000640 1 -- 172.21.15.63:0/3843335954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557043df5620 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa269a00640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa269000640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440b8a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ee9c0 tx=0x7fa254009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7055e3266c47f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa252a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540bcb70 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fa254030b90 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2540bd910 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa2540bcd50 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa26e97d740 10 client.27074 fetch_fsmap learned FSMap version 50 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.671+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5570440be580 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2540bd910 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fa2540eec40 con 0x5570440b09b0 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 10 client.27074 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 10 client.27074 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 20 client.27074 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5570440cb900 0x5570440cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 4 client.27074 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55704402a8b0 con 0x5570440cb900 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.672+0000 7fa263e00640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5570440cb900 0x5570440cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.673+0000 7fa263e00640 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5570440cb900 0x5570440cdd10 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f5e44550e394c1c server_cookie=4271af72e1a59959 in_seq=0 out_seq=0 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.673+0000 7fa252a00640 10 client.27074 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa252a00640 1 -- 172.21.15.63:0/3843335954 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55704402a8b0 con 0x5570440cb900 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa252a00640 10 client.27074 handle_command_reply: tid=0 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 client.27074 shutdown 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 msgr2=0x5570440b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440b8a50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa2540ee9c0 tx=0x7fa254009500 comp rx=0 tx=0).stop 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 shutdown_connections 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5570440cb900 0x5570440cdd10 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 --2- 172.21.15.63:0/3843335954 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440b8a50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa252a00640 0 client.27074 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 >> 172.21.15.63:0/3843335954 conn(0x5570440acec0 msgr2=0x5570440b2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 shutdown_connections 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 1 -- 172.21.15.63:0/3843335954 wait complete. 2024-04-25T19:02:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.674+0000 7fa26e97d740 20 client.27074 trim_cache size 0 max 16384 2024-04-25T19:02:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 Processor -- start 2024-04-25T19:02:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 -- start start 2024-04-25T19:02:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 0x5570440c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.693 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440d35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641b26d0 con 0x5570440bbb90 2024-04-25T19:02:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641afee0 con 0x5570440b12d0 2024-04-25T19:02:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.691+0000 7fa26e97d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641add50 con 0x5570440b09b0 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa263e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440d35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 0x5570440c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa263e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:48216/0 (socket says 172.21.15.63:48216) 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa263e00640 1 -- 172.21.15.63:0/1056230323 learned_addr learned my addr 172.21.15.63:0/1056230323 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 -- 172.21.15.63:0/1056230323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 msgr2=0x5570440b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 -- 172.21.15.63:0/1056230323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 msgr2=0x5570440d35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440d35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 -- 172.21.15.63:0/1056230323 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570440accf0 con 0x5570440b09b0 2024-04-25T19:02:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa263e00640 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:02:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa269000640 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 0x5570440c73b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa2540b9580 tx=0x7fa2540bb720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46414a40b7dfc6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.692+0000 7fa259600640 1 -- 172.21.15.63:0/1056230323 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540bcfe0 con 0x5570440b09b0 2024-04-25T19:02:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa259600640 1 -- 172.21.15.63:0/1056230323 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2540f4b90 con 0x5570440b09b0 2024-04-25T19:02:59.696 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa259600640 1 -- 172.21.15.63:0/1056230323 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540b8d80 con 0x5570440b09b0 2024-04-25T19:02:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 -- 172.21.15.63:0/1056230323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 msgr2=0x5570440c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 0x5570440c73b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa2540b9580 tx=0x7fa2540bb720 comp rx=0 tx=0).stop 2024-04-25T19:02:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 -- 172.21.15.63:0/1056230323 shutdown_connections 2024-04-25T19:02:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b12d0 0x5570440b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.699 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440d35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 --2- 172.21.15.63:0/1056230323 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b09b0 0x5570440c73b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 -- 172.21.15.63:0/1056230323 >> 172.21.15.63:0/1056230323 conn(0x5570440acec0 msgr2=0x5570440ac020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 -- 172.21.15.63:0/1056230323 shutdown_connections 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.693+0000 7fa26e97d740 1 -- 172.21.15.63:0/1056230323 wait complete. 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.694+0000 7fa26e97d740 1 -- start start 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.694+0000 7fa26e97d740 1 Processor -- start 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.694+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.694+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.694+0000 7fa26e97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440c1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa26e97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641ae3a0 con 0x5570440bbb90 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa26e97d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641af810 con 0x5570440b09b0 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa26e97d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641aea70 con 0x5570440b12d0 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa26e97d740 10 client.0 _ll_register_callbacks cb 0x7fa2540f9360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:02:59.700 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa269a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48554/0 (socket says 172.21.15.63:48554) 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 -- 172.21.15.63:0/2123747696 learned_addr learned my addr 172.21.15.63:0/2123747696 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa269000640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 -- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 msgr2=0x5570440c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440c3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 -- 172.21.15.63:0/2123747696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 msgr2=0x5570440c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.695+0000 7fa263e00640 1 -- 172.21.15.63:0/2123747696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570440c1b80 con 0x5570440b12d0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa263e00640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa26410eb80 tx=0x7fa24c01f9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f5666eda5d7ddb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa252a00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c005220 con 0x5570440b12d0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2540f4b90 con 0x5570440b12d0 2024-04-25T19:02:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fa24c0053c0 con 0x5570440b12d0 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fa24c0055c0 con 0x5570440b12d0 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa26e97d740 10 client.27091 fetch_fsmap learned FSMap version 50 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.696+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x557044131e00 con 0x5570440b12d0 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2540f4b90 con 0x5570440b12d0 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fa24c0278d0 con 0x5570440b12d0 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 10 client.27091 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 10 client.27091 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 20 client.27091 populate_metadata read hostname 'smithi063' 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557044132640 0x557044134a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:02:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 4 client.27091 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:02:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557044024320 con 0x557044132640 2024-04-25T19:02:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.697+0000 7fa269000640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557044132640 0x557044134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:02:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.698+0000 7fa269000640 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557044132640 0x557044134a60 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f2fb0524b058dc4 server_cookie=2bc1b9a0c800cf9 in_seq=0 out_seq=0 2024-04-25T19:02:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.698+0000 7fa252a00640 10 client.27091 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa252a00640 1 -- 172.21.15.63:0/2123747696 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557044024320 con 0x557044132640 2024-04-25T19:02:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa252a00640 10 client.27091 handle_command_reply: tid=0 2024-04-25T19:02:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 client.27091 shutdown 2024-04-25T19:02:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 msgr2=0x5570440c0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa26410eb80 tx=0x7fa24c01f9a0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 shutdown_connections 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x557044132640 0x557044134a60 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5570440bbb90 0x5570440c1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5570440b12d0 0x5570440c0f50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 --2- 172.21.15.63:0/2123747696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5570440b09b0 0x5570440c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa252a00640 0 client.27091 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.747+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 >> 172.21.15.63:0/2123747696 conn(0x5570440acec0 msgr2=0x5570440ac020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.748+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 shutdown_connections 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.748+0000 7fa26e97d740 1 -- 172.21.15.63:0/2123747696 wait complete. 2024-04-25T19:02:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.748+0000 7fa26e97d740 20 client.27091 trim_cache size 0 max 16384 2024-04-25T19:02:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:02:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:02:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:02:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:02:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 msgr2=0x7fa2480a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 0x7fa2480a61d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009dc0 tx=0x7fa25c009260 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 msgr2=0x7fa2641a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa25402acb0 tx=0x7fa254003b80 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 shutdown_connections 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa2480a3d10 0x7fa2480a61d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa26410e6d0 0x7fa26419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa2641014d0 0x7fa2641a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 --2- 172.21.15.63:0/253856309 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa264100b20 0x7fa2641a0460 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 >> 172.21.15.63:0/253856309 conn(0x7fa2640fc770 msgr2=0x7fa2640feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 shutdown_connections 2024-04-25T19:02:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:02:59.749+0000 7fa26a400640 1 -- 172.21.15.63:0/253856309 wait complete. 2024-04-25T19:03:00.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 18.23 MB/sec execute 58 sec latency 26331.302 ms 2024-04-25T19:03:00.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:00 smithi070 ceph-mon[32820]: pgmap v518: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 7.1 MiB/s wr, 268 op/s 2024-04-25T19:03:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:00 smithi114 ceph-mon[32988]: pgmap v518: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 7.1 MiB/s wr, 268 op/s 2024-04-25T19:03:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:00 smithi063 ceph-mon[28006]: pgmap v518: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 7.1 MiB/s wr, 268 op/s 2024-04-25T19:03:01.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 17.92 MB/sec execute 59 sec latency 27331.474 ms 2024-04-25T19:03:01.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:03:02.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 17.62 MB/sec execute 60 sec latency 28331.579 ms 2024-04-25T19:03:02.319 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:03:02.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:02 smithi070 ceph-mon[32820]: pgmap v519: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:02 smithi114 ceph-mon[32988]: pgmap v519: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:02.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:02 smithi063 ceph-mon[28006]: pgmap v519: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:03.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 17.33 MB/sec execute 61 sec latency 29331.663 ms 2024-04-25T19:03:04.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 17.05 MB/sec execute 62 sec latency 30331.754 ms 2024-04-25T19:03:04.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:04 smithi070 ceph-mon[32820]: pgmap v520: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:04.722 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:04 smithi063 ceph-mon[28006]: pgmap v520: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:04 smithi114 ceph-mon[32988]: pgmap v520: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.6 MiB/s wr, 166 op/s 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 -- 172.21.15.63:0/2241521193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 msgr2=0x7f0f24108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/2241521193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f24108e10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009990 tx=0x7f0f14009ca0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 -- 172.21.15.63:0/2241521193 shutdown_connections 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/2241521193 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406bac0 0x7f0f2410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/2241521193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f24108e10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/2241521193 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406b120 0x7f0f2406b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.061+0000 7f0f2aa00700 1 -- 172.21.15.63:0/2241521193 >> 172.21.15.63:0/2241521193 conn(0x7f0f240788b0 msgr2=0x7f0f24078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.062+0000 7f0f2aa00700 1 -- 172.21.15.63:0/2241521193 shutdown_connections 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.062+0000 7f0f2aa00700 1 -- 172.21.15.63:0/2241521193 wait complete. 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 Processor -- start 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 -- start start 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406bac0 0x7f0f2419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f2419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24112c20 con 0x7f0f24108a30 2024-04-25T19:03:05.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24112aa0 con 0x7f0f2406b120 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f2aa00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24112da0 con 0x7f0f2406bac0 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f23e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f23400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406bac0 0x7f0f2419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f28a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f2419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f23e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:48252/0 (socket says 172.21.15.63:48252) 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.063+0000 7f0f23e00700 1 -- 172.21.15.63:0/3003978222 learned_addr learned my addr 172.21.15.63:0/3003978222 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 -- 172.21.15.63:0/3003978222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406bac0 msgr2=0x7f0f2419b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406bac0 0x7f0f2419b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 -- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 msgr2=0x7f0f2419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f2419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 -- 172.21.15.63:0/3003978222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f241a0360 con 0x7f0f2406b120 2024-04-25T19:03:05.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f28a00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f2419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:05.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.064+0000 7f0f23e00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f140057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:05.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.065+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14027070 con 0x7f0f2406b120 2024-04-25T19:03:05.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.065+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f241a05f0 con 0x7f0f2406b120 2024-04-25T19:03:05.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.065+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f14022470 con 0x7f0f2406b120 2024-04-25T19:03:05.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.065+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14017650 con 0x7f0f2406b120 2024-04-25T19:03:05.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.065+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f241a7ed0 con 0x7f0f2406b120 2024-04-25T19:03:05.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.066+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f24104170 con 0x7f0f2406b120 2024-04-25T19:03:05.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.068+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0f1402a070 con 0x7f0f2406b120 2024-04-25T19:03:05.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.070+0000 7f0f20c00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 0x7f0f080a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:05.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.070+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7f0f140d14d0 con 0x7f0f2406b120 2024-04-25T19:03:05.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.070+0000 7f0f23400700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 0x7f0f080a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:05.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.071+0000 7f0f23400700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 0x7f0f080a62f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f2410fd40 tx=0x7f0f10009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:05.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.072+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f14013070 con 0x7f0f2406b120 2024-04-25T19:03:05.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 16.78 MB/sec execute 63 sec latency 31331.853 ms 2024-04-25T19:03:05.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.375+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f24069730 con 0x7f0f2406b120 2024-04-25T19:03:05.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.376+0000 7f0f20c00700 1 -- 172.21.15.63:0/3003978222 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+11874 (secure 0 0 0) 0x7f0f24069730 con 0x7f0f2406b120 2024-04-25T19:03:05.379 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:03:05.380 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":50,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:02:50.936225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26443":{"gid":26443,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/2546666842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":2546666842},{"type":"v1","addr":"172.21.15.114:6835","nonce":2546666842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T19:03:05.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.382+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 msgr2=0x7f0f080a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:05.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.382+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 0x7f0f080a62f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0f2410fd40 tx=0x7f0f10009260 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.382+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 msgr2=0x7f0f2419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.382+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0f14000c00 tx=0x7f0f140057c0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 shutdown_connections 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0f080a3e30 0x7f0f080a62f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f24108a30 0x7f0f2419fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f2406b120 0x7f0f2419b350 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 --2- 172.21.15.63:0/3003978222 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0f2406bac0 0x7f0f2419b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 >> 172.21.15.63:0/3003978222 conn(0x7f0f240788b0 msgr2=0x7f0f240ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 shutdown_connections 2024-04-25T19:03:05.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:05.383+0000 7f0f2aa00700 1 -- 172.21.15.63:0/3003978222 wait complete. 2024-04-25T19:03:05.387 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 50 2024-04-25T19:03:05.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:05 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3003978222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:03:05.794 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:05 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3003978222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:03:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:05 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3003978222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:03:06.092 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 16.52 MB/sec execute 64 sec latency 32331.962 ms 2024-04-25T19:03:06.276 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id aeec7e7c --release --await' 2024-04-25T19:03:06.276 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id aeec7e7c --release --await 2024-04-25T19:03:06.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:06 smithi070 ceph-mon[32820]: pgmap v521: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 7.3 MiB/s wr, 296 op/s 2024-04-25T19:03:06.737 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:03:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:06 smithi114 ceph-mon[32988]: pgmap v521: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 7.3 MiB/s wr, 296 op/s 2024-04-25T19:03:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:06 smithi063 ceph-mon[28006]: pgmap v521: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 7.3 MiB/s wr, 296 op/s 2024-04-25T19:03:07.093 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 16.26 MB/sec execute 65 sec latency 33333.145 ms 2024-04-25T19:03:08.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 16.02 MB/sec execute 66 sec latency 34333.270 ms 2024-04-25T19:03:08.605 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:08 smithi063 ceph-mon[28006]: pgmap v522: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.9 MiB/s wr, 217 op/s 2024-04-25T19:03:08.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:08 smithi070 ceph-mon[32820]: pgmap v522: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.9 MiB/s wr, 217 op/s 2024-04-25T19:03:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:08 smithi114 ceph-mon[32988]: pgmap v522: 97 pgs: 97 active+clean; 677 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 4.9 MiB/s wr, 217 op/s 2024-04-25T19:03:09.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71426 15.78 MB/sec execute 67 sec latency 35333.400 ms 2024-04-25T19:03:09.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.150+0000 7fd34d000700 1 -- 172.21.15.63:0/3808624372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348109fc0 msgr2=0x7fd348111b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.150+0000 7fd34d000700 1 --2- 172.21.15.63:0/3808624372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348109fc0 0x7fd348111b90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd34000b0d0 tx=0x7fd34000b3e0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 -- 172.21.15.63:0/3808624372 shutdown_connections 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 --2- 172.21.15.63:0/3808624372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348104ee0 0x7fd3481052c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 --2- 172.21.15.63:0/3808624372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348109fc0 0x7fd348111b90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 --2- 172.21.15.63:0/3808624372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348105890 0x7fd3481098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 -- 172.21.15.63:0/3808624372 >> 172.21.15.63:0/3808624372 conn(0x7fd348100770 msgr2=0x7fd348102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 -- 172.21.15.63:0/3808624372 shutdown_connections 2024-04-25T19:03:09.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.151+0000 7fd34d000700 1 -- 172.21.15.63:0/3808624372 wait complete. 2024-04-25T19:03:09.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 Processor -- start 2024-04-25T19:03:09.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 -- start start 2024-04-25T19:03:09.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 0x7fd34819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348109fc0 0x7fd34819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd348114ec0 con 0x7fd348104ee0 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd348114d40 con 0x7fd348109fc0 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.152+0000 7fd34d000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd348115040 con 0x7fd348105890 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47146/0 (socket says 172.21.15.63:47146) 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 -- 172.21.15.63:0/887463821 learned_addr learned my addr 172.21.15.63:0/887463821 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345400700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 0x7fd34819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd346800700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348109fc0 0x7fd34819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 -- 172.21.15.63:0/887463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 msgr2=0x7fd34819b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 0x7fd34819b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 -- 172.21.15.63:0/887463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348109fc0 msgr2=0x7fd34819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348109fc0 0x7fd34819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345e00700 1 -- 172.21.15.63:0/887463821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3481a0300 con 0x7fd348104ee0 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.153+0000 7fd345400700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 0x7fd34819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd345e00700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd33800e9c0 tx=0x7fd33800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd33ea00700 1 -- 172.21.15.63:0/887463821 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33800cbd0 con 0x7fd348104ee0 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd33ea00700 1 -- 172.21.15.63:0/887463821 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd33800ee60 con 0x7fd348104ee0 2024-04-25T19:03:09.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3481a05f0 con 0x7fd348104ee0 2024-04-25T19:03:09.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd33ea00700 1 -- 172.21.15.63:0/887463821 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380055c0 con 0x7fd348104ee0 2024-04-25T19:03:09.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.154+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3481a7e70 con 0x7fd348104ee0 2024-04-25T19:03:09.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.157+0000 7fd33ea00700 1 -- 172.21.15.63:0/887463821 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fd338004750 con 0x7fd348104ee0 2024-04-25T19:03:09.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd33ea00700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 0x7fd32c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 Processor -- start 2024-04-25T19:03:09.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 -- start start 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd33ea00700 1 -- 172.21.15.63:0/887463821 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fd3380d0ab0 con 0x7fd348104ee0 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd345400700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 0x7fd32c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 0x56507da99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da9c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56507d9e1f90 con 0x56507da9a4d0 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56507d988830 con 0x56507da8ed40 2024-04-25T19:03:09.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.159+0000 7fd35793db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56507d9ddf50 con 0x56507da8f660 2024-04-25T19:03:09.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59032/0 (socket says 172.21.15.63:59032) 2024-04-25T19:03:09.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 -- 172.21.15.63:0/1985282656 learned_addr learned my addr 172.21.15.63:0/1985282656 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd346800700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da9c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345400700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 0x56507da99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 -- 172.21.15.63:0/1985282656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 msgr2=0x56507da99f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 0x56507da99f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 -- 172.21.15.63:0/1985282656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 msgr2=0x56507da9c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da9c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345e00700 1 -- 172.21.15.63:0/1985282656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507d97e780 con 0x56507da8ed40 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345400700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 0x56507da99f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.160+0000 7fd345400700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 0x7fd32c0a62f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd34810b4b0 tx=0x7fd334011820 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.161+0000 7fd345e00700 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c5b0 tx=0x7fd33809c5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0d8990714ee2753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.161+0000 7fd333400700 1 -- 172.21.15.63:0/1985282656 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33809c9b0 con 0x56507da8ed40 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.161+0000 7fd333400700 1 -- 172.21.15.63:0/1985282656 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3380a0d60 con 0x56507da8ed40 2024-04-25T19:03:09.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.161+0000 7fd333400700 1 -- 172.21.15.63:0/1985282656 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3380a5190 con 0x56507da8ed40 2024-04-25T19:03:09.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 -- 172.21.15.63:0/1985282656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 msgr2=0x56507da8f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c5b0 tx=0x7fd33809c5c0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 -- 172.21.15.63:0/1985282656 shutdown_connections 2024-04-25T19:03:09.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da9c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8ed40 0x56507da8f120 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 --2- 172.21.15.63:0/1985282656 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8f660 0x56507da99f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 -- 172.21.15.63:0/1985282656 >> 172.21.15.63:0/1985282656 conn(0x56507d9be050 msgr2=0x56507da8b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 -- 172.21.15.63:0/1985282656 shutdown_connections 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.162+0000 7fd35793db80 1 -- 172.21.15.63:0/1985282656 wait complete. 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 -- start start 2024-04-25T19:03:09.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 Processor -- start 2024-04-25T19:03:09.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da96f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da97490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56507da9cf60 con 0x56507da9a4d0 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x56507d99b960 con 0x56507da8f660 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.163+0000 7fd35793db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x56507da9d0e0 con 0x56507da8ed40 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd35793db80 10 client.0 _ll_register_callbacks cb 0x56507d97a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd346800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da97490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58422/0 (socket says 172.21.15.63:58422) 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da96f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345e00700 1 -- 172.21.15.63:0/115705136 learned_addr learned my addr 172.21.15.63:0/115705136 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd346800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da97490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47164/0 (socket says 172.21.15.63:47164) 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 -- 172.21.15.63:0/115705136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 msgr2=0x56507da99c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 -- 172.21.15.63:0/115705136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 msgr2=0x56507da97490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da97490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345400700 1 -- 172.21.15.63:0/115705136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507d92f9d0 con 0x56507da8f660 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.164+0000 7fd345e00700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd345400700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da96f50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd34806b500 tx=0x7fd3340047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e77a4ab67dbb9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd327e00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334011e60 con 0x56507da8f660 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fd334004320 con 0x56507da8f660 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3380a0d60 con 0x56507da8f660 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd334017040 con 0x56507da8f660 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd35793db80 10 client.27098 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:09.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.165+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56507d978bf0 con 0x56507da8f660 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3380a0d60 con 0x56507da8f660 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fd33402d030 con 0x56507da8f660 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 10 client.27098 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 10 client.27098 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 20 client.27098 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507daa9d80 0x56507daac170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 4 client.27098 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56507d9ef780 con 0x56507daa9d80 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.166+0000 7fd345e00700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507daa9d80 0x56507daac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.167+0000 7fd345e00700 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507daa9d80 0x56507daac170 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac0d7e9b23b2a77a server_cookie=e181b5440f43eb1b in_seq=0 out_seq=0 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.167+0000 7fd327e00700 10 client.27098 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd327e00700 1 -- 172.21.15.63:0/115705136 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56507d9ef780 con 0x56507daa9d80 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd327e00700 10 client.27098 handle_command_reply: tid=0 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 client.27098 shutdown 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 msgr2=0x56507da96f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da96f50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd34806b500 tx=0x7fd3340047c0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 shutdown_connections 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507daa9d80 0x56507daac170 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da99c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da97490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 --2- 172.21.15.63:0/115705136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da96f50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 >> 172.21.15.63:0/115705136 conn(0x56507d9be050 msgr2=0x56507da8ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd327e00700 0 client.27098 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 shutdown_connections 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 1 -- 172.21.15.63:0/115705136 wait complete. 2024-04-25T19:03:09.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.168+0000 7fd35793db80 20 client.27098 trim_cache size 0 max 16384 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 Processor -- start 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 -- start start 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da97580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da927a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da98e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481b1fd0 con 0x56507da9a4d0 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481af7a0 con 0x56507da8f660 2024-04-25T19:03:09.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.190+0000 7fd35793db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481ad610 con 0x56507da8ed40 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd346800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da98e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd346800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da98e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47168/0 (socket says 172.21.15.63:47168) 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd346800700 1 -- 172.21.15.63:0/3889726008 learned_addr learned my addr 172.21.15.63:0/3889726008 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345e00700 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da97580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 -- 172.21.15.63:0/3889726008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 msgr2=0x56507da97580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da97580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 -- 172.21.15.63:0/3889726008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 msgr2=0x56507da98e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da98e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.191+0000 7fd345400700 1 -- 172.21.15.63:0/3889726008 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507da8bd20 con 0x56507da8f660 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd345400700 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da927a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd33401e820 tx=0x7fd334013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=196d50dc49fa90f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd332a00700 1 -- 172.21.15.63:0/3889726008 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334009240 con 0x56507da8f660 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd332a00700 1 -- 172.21.15.63:0/3889726008 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd334019050 con 0x56507da8f660 2024-04-25T19:03:09.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd332a00700 1 -- 172.21.15.63:0/3889726008 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334003d30 con 0x56507da8f660 2024-04-25T19:03:09.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd35793db80 1 -- 172.21.15.63:0/3889726008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 msgr2=0x56507da927a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.192+0000 7fd35793db80 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da927a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd33401e820 tx=0x7fd334013000 comp rx=0 tx=0).stop 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 -- 172.21.15.63:0/3889726008 shutdown_connections 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507da97580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da9a4d0 0x56507da98e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 --2- 172.21.15.63:0/3889726008 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da8f660 0x56507da927a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 -- 172.21.15.63:0/3889726008 >> 172.21.15.63:0/3889726008 conn(0x56507d9be050 msgr2=0x56507da98780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 -- 172.21.15.63:0/3889726008 shutdown_connections 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 -- 172.21.15.63:0/3889726008 wait complete. 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 -- start start 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.193+0000 7fd35793db80 1 Processor -- start 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507daa1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 0x56507da9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481adc60 con 0x56507da8f660 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481af0d0 con 0x56507da9a4d0 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fd3481ae330 con 0x56507da8ed40 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd35793db80 10 client.0 _ll_register_callbacks cb 0x56507d97a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd345400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd345400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47176/0 (socket says 172.21.15.63:47176) 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd345400700 1 -- 172.21.15.63:0/2606147180 learned_addr learned my addr 172.21.15.63:0/2606147180 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd346800700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 0x56507da9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.194+0000 7fd345e00700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507daa1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 -- 172.21.15.63:0/2606147180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 msgr2=0x56507daa1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507daa1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 -- 172.21.15.63:0/2606147180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 msgr2=0x56507daa24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 -- 172.21.15.63:0/2606147180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507da8fce0 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd345400700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd346800700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 0x56507da9f460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd34000b0d0 tx=0x7fd340004c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f91fd1944b75641a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd327e00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3400076e0 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd334019050 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.195+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(119..119 src has 1..119) ==== 8587+0+0 (secure 0 0 0) 0x7fd340007880 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fd340007a80 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 10 client.27104 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x56507db11bf0 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd334019050 con 0x56507da9a4d0 2024-04-25T19:03:09.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fd34002e030 con 0x56507da9a4d0 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 10 client.27104 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 10 client.27104 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 20 client.27104 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507db12390 0x56507db147b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 4 client.27104 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "aeec7e7c", "release": true, "await": true}] 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.196+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "aeec7e7c", "release": true, "await": true}) -- 0x56507d9e9820 con 0x56507db12390 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.197+0000 7fd345e00700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507db12390 0x56507db147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.197+0000 7fd345e00700 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507db12390 0x56507db147b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69c47c9d9e9252c3 server_cookie=deaf6d34a39b253b in_seq=0 out_seq=0 2024-04-25T19:03:09.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.197+0000 7fd327e00700 10 client.27104 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:09.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.222+0000 7fd327e00700 1 -- 172.21.15.63:0/2606147180 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+590 (crc 0 0 0) 0x56507d9e9820 con 0x56507db12390 2024-04-25T19:03:09.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.222+0000 7fd327e00700 10 client.27104 handle_command_reply: tid=0 2024-04-25T19:03:09.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.222+0000 7fd35793db80 1 client.27104 shutdown 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 msgr2=0x56507da9f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 0x56507da9f460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fd34000b0d0 tx=0x7fd340004c40 comp rx=0 tx=0).stop 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 shutdown_connections 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56507db12390 0x56507db147b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x56507da8ed40 0x56507daa1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x56507da9a4d0 0x56507da9f460 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 --2- 172.21.15.63:0/2606147180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56507da8f660 0x56507daa24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 >> 172.21.15.63:0/2606147180 conn(0x56507d9be050 msgr2=0x56507da92620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd327e00700 0 client.27104 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 shutdown_connections 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 1 -- 172.21.15.63:0/2606147180 wait complete. 2024-04-25T19:03:09.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.223+0000 7fd35793db80 20 client.27104 trim_cache size 0 max 16384 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 50, 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 48, 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "aeec7e7c": { 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 48, 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T19:03:09.227 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 86.5, 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RELEASED", 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:03:09.228 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:03:09.229 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:03:09.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.224+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 msgr2=0x7fd32c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.224+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 0x7fd32c0a62f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd34810b4b0 tx=0x7fd334011820 comp rx=0 tx=0).stop 2024-04-25T19:03:09.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.224+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 msgr2=0x7fd34819b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.224+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd33800e9c0 tx=0x7fd33800ecd0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 shutdown_connections 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fd32c0a3e30 0x7fd32c0a62f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd348109fc0 0x7fd34819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd348104ee0 0x7fd34819b2f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 --2- 172.21.15.63:0/887463821 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fd348105890 0x7fd34819b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 >> 172.21.15.63:0/887463821 conn(0x7fd348100770 msgr2=0x7fd348101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 shutdown_connections 2024-04-25T19:03:09.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:03:09.225+0000 7fd34d000700 1 -- 172.21.15.63:0/887463821 wait complete. 2024-04-25T19:03:10.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 71737 15.76 MB/sec execute 68 sec latency 35499.626 ms 2024-04-25T19:03:10.291 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'aeec7e7c', seconds elapsed: 44.5 2024-04-25T19:03:10.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:10 smithi063 ceph-mon[28006]: pgmap v523: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-04-25T19:03:10.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:10 smithi063 ceph-mon[28006]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T19:03:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:10 smithi070 ceph-mon[32820]: pgmap v523: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-04-25T19:03:10.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:10 smithi070 ceph-mon[32820]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T19:03:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:10 smithi114 ceph-mon[32988]: pgmap v523: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 5.7 MiB/s wr, 262 op/s 2024-04-25T19:03:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:10 smithi114 ceph-mon[32988]: osdmap e120: 12 total, 12 up, 12 in 2024-04-25T19:03:11.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 72174 15.79 MB/sec execute 69 sec latency 100.207 ms 2024-04-25T19:03:12.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 72876 16.15 MB/sec execute 70 sec latency 58.071 ms 2024-04-25T19:03:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:12 smithi070 ceph-mon[32820]: pgmap v525: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:12.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:12 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:12 smithi114 ceph-mon[32988]: pgmap v525: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:12 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:12 smithi063 ceph-mon[28006]: pgmap v525: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:12.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:12 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.935+0000 7fb4e7e00640 1 -- 172.21.15.63:0/1612138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e0106680 msgr2=0x7fb4e0106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.935+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/1612138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e0106680 0x7fb4e0106a60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400aa40 tx=0x7fb4d402b760 comp rx=0 tx=0).stop 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 -- 172.21.15.63:0/1612138223 shutdown_connections 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/1612138223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0100f80 0x7fb4e010d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/1612138223 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e01005c0 0x7fb4e0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/1612138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e0106680 0x7fb4e0106a60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 -- 172.21.15.63:0/1612138223 >> 172.21.15.63:0/1612138223 conn(0x7fb4e00773e0 msgr2=0x7fb4e00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 -- 172.21.15.63:0/1612138223 shutdown_connections 2024-04-25T19:03:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.936+0000 7fb4e7e00640 1 -- 172.21.15.63:0/1612138223 wait complete. 2024-04-25T19:03:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.937+0000 7fb4e7e00640 1 Processor -- start 2024-04-25T19:03:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.937+0000 7fb4e7e00640 1 -- start start 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 0x7fb4e01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0106680 0x7fb4e019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0110000 con 0x7fb4e0106680 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e010fe80 con 0x7fb4e01005c0 2024-04-25T19:03:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e7e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e0110180 con 0x7fb4e0100f80 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 0x7fb4e01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 0x7fb4e01a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58478/0 (socket says 172.21.15.63:58478) 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59068/0 (socket says 172.21.15.63:59068) 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e4a00640 1 -- 172.21.15.63:0/864931360 learned_addr learned my addr 172.21.15.63:0/864931360 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0106680 0x7fb4e019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.938+0000 7fb4e5400640 1 -- 172.21.15.63:0/864931360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 msgr2=0x7fb4e01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.941 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e5400640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 0x7fb4e01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e5400640 1 -- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0106680 msgr2=0x7fb4e019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e5400640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0106680 0x7fb4e019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e5400640 1 -- 172.21.15.63:0/864931360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e019af70 con 0x7fb4e01005c0 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e5400640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb4d402bc80 tx=0x7fb4d4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4de000640 1 -- 172.21.15.63:0/864931360 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d403f070 con 0x7fb4e01005c0 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4de000640 1 -- 172.21.15.63:0/864931360 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d4030c70 con 0x7fb4e01005c0 2024-04-25T19:03:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e019b200 con 0x7fb4e01005c0 2024-04-25T19:03:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4de000640 1 -- 172.21.15.63:0/864931360 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4036040 con 0x7fb4e01005c0 2024-04-25T19:03:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.939+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e01a7340 con 0x7fb4e01005c0 2024-04-25T19:03:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.943+0000 7fb4eac19740 1 Processor -- start 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 -- start start 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144cfbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144d06a00 0x558144d08df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558144c61900 con 0x558144cfb820 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4de000640 1 -- 172.21.15.63:0/864931360 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fb4d4042070 con 0x7fb4e01005c0 2024-04-25T19:03:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558144c07ef0 con 0x558144cfc140 2024-04-25T19:03:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4eac19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558144c5d8c0 con 0x558144d06a00 2024-04-25T19:03:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.947 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.944+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59084/0 (socket says 172.21.15.63:59084) 2024-04-25T19:03:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.945+0000 7fb4e5400640 1 -- 172.21.15.63:0/2072916658 learned_addr learned my addr 172.21.15.63:0/2072916658 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.945+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144d06a00 0x558144d08df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.945+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144cfbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5400640 1 -- 172.21.15.63:0/2072916658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144d06a00 msgr2=0x558144d08df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144d06a00 0x558144d08df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5400640 1 -- 172.21.15.63:0/2072916658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 msgr2=0x558144cfbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144cfbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5400640 1 -- 172.21.15.63:0/2072916658 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558144974210 con 0x558144cfc140 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.946+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144cfbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:12.949 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.947+0000 7fb4de000640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 0x7fb4c80a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.947+0000 7fb4de000640 1 -- 172.21.15.63:0/864931360 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8631+0+0 (secure 0 0 0) 0x7fb4d40f0b20 con 0x7fb4e01005c0 2024-04-25T19:03:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.947+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 0x7fb4c80a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.947+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40ee170 tx=0x7fb4d40f4df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50b173a7217b3ca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.948+0000 7fb4d2600640 1 -- 172.21.15.63:0/2072916658 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4049020 con 0x558144cfc140 2024-04-25T19:03:12.950 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.948+0000 7fb4d2600640 1 -- 172.21.15.63:0/2072916658 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d40f5040 con 0x558144cfc140 2024-04-25T19:03:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.948+0000 7fb4d2600640 1 -- 172.21.15.63:0/2072916658 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40bc8d0 con 0x558144cfc140 2024-04-25T19:03:12.951 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.949+0000 7fb4eac19740 1 -- 172.21.15.63:0/2072916658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 msgr2=0x558144d064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.949+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40ee170 tx=0x7fb4d40f4df0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 0x7fb4c80a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009fd0 tx=0x7fb4d8009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 -- 172.21.15.63:0/2072916658 shutdown_connections 2024-04-25T19:03:12.952 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d064c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144d06a00 0x558144d08df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2072916658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144cfbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 -- 172.21.15.63:0/2072916658 >> 172.21.15.63:0/2072916658 conn(0x558144cf7d30 msgr2=0x558144cf8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 -- 172.21.15.63:0/2072916658 shutdown_connections 2024-04-25T19:03:12.953 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.950+0000 7fb4eac19740 1 -- 172.21.15.63:0/2072916658 wait complete. 2024-04-25T19:03:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.951+0000 7fb4eac19740 1 -- start start 2024-04-25T19:03:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.951+0000 7fb4eac19740 1 Processor -- start 2024-04-25T19:03:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144d03990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.954 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d04410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558144d093f0 con 0x558144cfb820 2024-04-25T19:03:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x558144c1b310 con 0x558144d06a00 2024-04-25T19:03:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x558144d09570 con 0x558144cfc140 2024-04-25T19:03:12.955 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4eac19740 10 client.0 _ll_register_callbacks cb 0x558144be9cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58486/0 (socket says 172.21.15.63:58486) 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4e4a00640 1 -- 172.21.15.63:0/1541699068 learned_addr learned my addr 172.21.15.63:0/1541699068 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4e5400640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144d03990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.952+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d04410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 -- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 msgr2=0x558144d04410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d04410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 -- 172.21.15.63:0/1541699068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 msgr2=0x558144d03990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144d03990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 -- 172.21.15.63:0/1541699068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558144a40620 con 0x558144cfc140 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d04410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00bf30 tx=0x7fb4cc012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d7a9eef04c70072 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.956 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:03:12.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.953+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc004230 con 0x558144cfc140 2024-04-25T19:03:12.958 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4d40f5040 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8631+0+0 (secure 0 0 0) 0x7fb4cc0043d0 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb4cc010760 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4eac19740 10 client.27127 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558144d098a0 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.954+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4d40f5040 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fb4cc01b030 con 0x558144cfc140 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 10 client.27127 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 10 client.27127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 20 client.27127 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d0add0 0x558144d0d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 4 client.27127 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558144c758b0 con 0x558144d0add0 2024-04-25T19:03:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4e5400640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d0add0 0x558144d0d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.955+0000 7fb4e5400640 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d0add0 0x558144d0d210 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62127e1bcfe339f server_cookie=dccfab9318d78620 in_seq=0 out_seq=0 2024-04-25T19:03:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.956+0000 7fb4c6a00640 10 client.27127 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.956+0000 7fb4c6a00640 1 -- 172.21.15.63:0/1541699068 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558144c758b0 con 0x558144d0add0 2024-04-25T19:03:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.956+0000 7fb4c6a00640 10 client.27127 handle_command_reply: tid=0 2024-04-25T19:03:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.956+0000 7fb4eac19740 1 client.27127 shutdown 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 msgr2=0x558144d03ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00bf30 tx=0x7fb4cc012820 comp rx=0 tx=0).stop 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 shutdown_connections 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d0add0 0x558144d0d210 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d04410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfc140 0x558144d03ed0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 --2- 172.21.15.63:0/1541699068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfb820 0x558144d03990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 >> 172.21.15.63:0/1541699068 conn(0x558144cf7d30 msgr2=0x558144cfdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4c6a00640 0 client.27127 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 shutdown_connections 2024-04-25T19:03:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 1 -- 172.21.15.63:0/1541699068 wait complete. 2024-04-25T19:03:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.957+0000 7fb4eac19740 20 client.27127 trim_cache size 0 max 16384 2024-04-25T19:03:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 Processor -- start 2024-04-25T19:03:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 -- start start 2024-04-25T19:03:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144cfe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144d06a00 0x558144d232a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c0011270 con 0x558144d06a00 2024-04-25T19:03:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c000ea80 con 0x558144cfc140 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.983+0000 7fb4eac19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c000c8f0 con 0x558144cfb820 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144cfe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59102/0 (socket says 172.21.15.63:59102) 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 -- 172.21.15.63:0/961791832 learned_addr learned my addr 172.21.15.63:0/961791832 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144cfe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:58498/0 (socket says 172.21.15.63:58498) 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144d06a00 0x558144d232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 -- 172.21.15.63:0/961791832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 msgr2=0x558144cfe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144cfe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 -- 172.21.15.63:0/961791832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144d06a00 msgr2=0x558144d232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144d06a00 0x558144d232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 -- 172.21.15.63:0/961791832 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558144cfbf70 con 0x558144cfc140 2024-04-25T19:03:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.984+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc003cc0 tx=0x7fb4cc001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa3aa911597ca65e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.985+0000 7fb4d1c00640 1 -- 172.21.15.63:0/961791832 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0037f0 con 0x558144cfc140 2024-04-25T19:03:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.985+0000 7fb4d1c00640 1 -- 172.21.15.63:0/961791832 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4cc003990 con 0x558144cfc140 2024-04-25T19:03:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.985+0000 7fb4d1c00640 1 -- 172.21.15.63:0/961791832 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc024420 con 0x558144cfc140 2024-04-25T19:03:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.986+0000 7fb4eac19740 1 -- 172.21.15.63:0/961791832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 msgr2=0x558144d22d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.986+0000 7fb4eac19740 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc003cc0 tx=0x7fb4cc001e70 comp rx=0 tx=0).stop 2024-04-25T19:03:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.986+0000 7fb4eac19740 1 -- 172.21.15.63:0/961791832 shutdown_connections 2024-04-25T19:03:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.986+0000 7fb4eac19740 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144cfc140 0x558144d22d60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.986+0000 7fb4eac19740 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144d06a00 0x558144d232a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.987+0000 7fb4eac19740 1 --2- 172.21.15.63:0/961791832 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144cfe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.987+0000 7fb4eac19740 1 -- 172.21.15.63:0/961791832 >> 172.21.15.63:0/961791832 conn(0x558144cf7d30 msgr2=0x558144cf6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.987+0000 7fb4eac19740 1 -- 172.21.15.63:0/961791832 shutdown_connections 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.987+0000 7fb4eac19740 1 -- 172.21.15.63:0/961791832 wait complete. 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.988+0000 7fb4eac19740 1 -- start start 2024-04-25T19:03:12.990 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.988+0000 7fb4eac19740 1 Processor -- start 2024-04-25T19:03:12.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144d75d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.991 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 0x558144d76290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c000cf40 con 0x558144cfc140 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c000e3b0 con 0x558144d06a00 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c000d610 con 0x558144cfb820 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4eac19740 10 client.0 _ll_register_callbacks cb 0x7fb4cc00bb40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4e5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4e5e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:59106/0 (socket says 172.21.15.63:59106) 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4e5e00640 1 -- 172.21.15.63:0/2797018798 learned_addr learned my addr 172.21.15.63:0/2797018798 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.989+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144d75d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 0x558144d76290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e5e00640 1 -- 172.21.15.63:0/2797018798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 msgr2=0x558144d75d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144d75d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e5e00640 1 -- 172.21.15.63:0/2797018798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 msgr2=0x558144d76290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 0x558144d76290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e5e00640 1 -- 172.21.15.63:0/2797018798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558144d229d0 con 0x558144d06a00 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.990+0000 7fb4e4a00640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 0x558144d76290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.991+0000 7fb4e5e00640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8012010 tx=0x7fb4d8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237ffd044178d6de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.991+0000 7fb4c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.991+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d803f060 con 0x558144d06a00 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.992+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4cc003990 con 0x558144d06a00 2024-04-25T19:03:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.992+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8631+0+0 (secure 0 0 0) 0x7fb4d8017030 con 0x558144d06a00 2024-04-25T19:03:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.992+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fb4d803f060 con 0x558144d06a00 2024-04-25T19:03:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.992+0000 7fb4eac19740 10 client.27122 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.992+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558144d7d640 con 0x558144d06a00 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.993+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4cc003990 con 0x558144d06a00 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.993+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7fb4d8039070 con 0x558144d06a00 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 10 client.27122 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 10 client.27122 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 20 client.27122 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d7de00 0x558144d80220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 4 client.27122 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:03:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558144c6f320 con 0x558144d7de00 2024-04-25T19:03:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d7de00 0x558144d80220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4e5400640 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d7de00 0x558144d80220 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8957eec0b299dd91 server_cookie=342f060ba5b05aa6 in_seq=0 out_seq=0 2024-04-25T19:03:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:12.994+0000 7fb4c6a00640 10 client.27122 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:13.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 73552 16.45 MB/sec execute 71 sec latency 80.266 ms 2024-04-25T19:03:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.141+0000 7fb4c6a00640 1 -- 172.21.15.63:0/2797018798 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558144c6f320 con 0x558144d7de00 2024-04-25T19:03:13.144 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.141+0000 7fb4c6a00640 10 client.27122 handle_command_reply: tid=0 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.141+0000 7fb4eac19740 1 client.27122 shutdown 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.141+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 msgr2=0x558144d222e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.141+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8012010 tx=0x7fb4d8004ab0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 shutdown_connections 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x558144d7de00 0x558144d80220 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x558144d06a00 0x558144d222e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558144cfc140 0x558144d76290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 --2- 172.21.15.63:0/2797018798 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x558144cfb820 0x558144d75d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 >> 172.21.15.63:0/2797018798 conn(0x558144cf7d30 msgr2=0x558144cf6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4c6a00640 0 client.27122 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 shutdown_connections 2024-04-25T19:03:13.145 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 1 -- 172.21.15.63:0/2797018798 wait complete. 2024-04-25T19:03:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.142+0000 7fb4eac19740 20 client.27122 trim_cache size 0 max 16384 2024-04-25T19:03:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:03:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:03:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:03:13.146 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.144+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 msgr2=0x7fb4c80a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.144+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 0x7fb4c80a6300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009fd0 tx=0x7fb4d8009500 comp rx=0 tx=0).stop 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.144+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 msgr2=0x7fb4e01a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.144+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb4d402bc80 tx=0x7fb4d4003c10 comp rx=0 tx=0).stop 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 shutdown_connections 2024-04-25T19:03:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fb4c80a3e40 0x7fb4c80a6300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb4e0106680 0x7fb4e019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fb4e0100f80 0x7fb4e01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 --2- 172.21.15.63:0/864931360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb4e01005c0 0x7fb4e01a05a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 >> 172.21.15.63:0/864931360 conn(0x7fb4e00773e0 msgr2=0x7fb4e0104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 shutdown_connections 2024-04-25T19:03:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:13.145+0000 7fb4e7e00640 1 -- 172.21.15.63:0/864931360 wait complete. 2024-04-25T19:03:13.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:13 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:13.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:13 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:13.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:13 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:03:13.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:13 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:13.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:13 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:13.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:13 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:03:13.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:13 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:03:13.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:13 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:03:13.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:13 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:03:14.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 74164 16.72 MB/sec execute 72 sec latency 94.797 ms 2024-04-25T19:03:14.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:14 smithi070 ceph-mon[32820]: pgmap v526: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:14 smithi114 ceph-mon[32988]: pgmap v526: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:14.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:14 smithi063 ceph-mon[28006]: pgmap v526: 97 pgs: 97 active+clean; 666 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 4.3 MiB/s wr, 209 op/s 2024-04-25T19:03:15.094 INFO:tasks.workunit.client.0.smithi063.stdout: 10 74785 17.06 MB/sec execute 73 sec latency 51.199 ms 2024-04-25T19:03:16.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 75349 17.05 MB/sec execute 74 sec latency 56.964 ms 2024-04-25T19:03:16.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:16 smithi070 ceph-mon[32820]: pgmap v527: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:16 smithi114 ceph-mon[32988]: pgmap v527: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:16.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:16 smithi063 ceph-mon[28006]: pgmap v527: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:17.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 76208 17.44 MB/sec execute 75 sec latency 60.076 ms 2024-04-25T19:03:18.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 77016 17.83 MB/sec execute 76 sec latency 80.240 ms 2024-04-25T19:03:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:18 smithi114 ceph-mon[32988]: pgmap v528: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:18 smithi063 ceph-mon[28006]: pgmap v528: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:18.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:18 smithi070 ceph-mon[32820]: pgmap v528: 97 pgs: 97 active+clean; 685 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.8 MiB/s wr, 348 op/s 2024-04-25T19:03:19.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 77671 18.06 MB/sec execute 77 sec latency 100.431 ms 2024-04-25T19:03:20.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 78235 18.26 MB/sec execute 78 sec latency 43.466 ms 2024-04-25T19:03:20.453 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:20 smithi070 ceph-mon[32820]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T19:03:20.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:20 smithi070 ceph-mon[32820]: pgmap v530: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.1 MiB/s wr, 565 op/s 2024-04-25T19:03:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:20 smithi114 ceph-mon[32988]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T19:03:20.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:20 smithi114 ceph-mon[32988]: pgmap v530: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.1 MiB/s wr, 565 op/s 2024-04-25T19:03:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:20 smithi063 ceph-mon[28006]: osdmap e121: 12 total, 12 up, 12 in 2024-04-25T19:03:20.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:20 smithi063 ceph-mon[28006]: pgmap v530: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.1 MiB/s wr, 565 op/s 2024-04-25T19:03:21.095 INFO:tasks.workunit.client.0.smithi063.stdout: 10 78840 18.35 MB/sec execute 79 sec latency 71.741 ms 2024-04-25T19:03:22.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 79611 18.62 MB/sec execute 80 sec latency 115.458 ms 2024-04-25T19:03:22.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:22 smithi070 ceph-mon[32820]: pgmap v531: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:22 smithi114 ceph-mon[32988]: pgmap v531: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:22.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:22 smithi063 ceph-mon[28006]: pgmap v531: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:23.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 80432 19.04 MB/sec execute 81 sec latency 50.933 ms 2024-04-25T19:03:24.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 81069 19.24 MB/sec execute 82 sec latency 48.463 ms 2024-04-25T19:03:24.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:24 smithi070 ceph-mon[32820]: pgmap v532: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:24 smithi114 ceph-mon[32988]: pgmap v532: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:24.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:24 smithi063 ceph-mon[28006]: pgmap v532: 97 pgs: 97 active+clean; 714 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 6.0 MiB/s wr, 559 op/s 2024-04-25T19:03:25.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 81582 19.38 MB/sec execute 83 sec latency 111.654 ms 2024-04-25T19:03:26.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 82234 19.44 MB/sec execute 84 sec latency 78.614 ms 2024-04-25T19:03:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 -- 172.21.15.63:0/3414864679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741072a0 msgr2=0x7f067410ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 --2- 172.21.15.63:0/3414864679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741072a0 0x7f067410ee70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f066c009ca0 tx=0x7f066c02a8c0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 -- 172.21.15.63:0/3414864679 shutdown_connections 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 --2- 172.21.15.63:0/3414864679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741072a0 0x7f067410ee70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 --2- 172.21.15.63:0/3414864679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0674102b80 0x7f0674106bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 --2- 172.21.15.63:0/3414864679 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741021d0 0x7f06741025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 -- 172.21.15.63:0/3414864679 >> 172.21.15.63:0/3414864679 conn(0x7f06740793e0 msgr2=0x7f06740797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 -- 172.21.15.63:0/3414864679 shutdown_connections 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.335+0000 7f067a200640 1 -- 172.21.15.63:0/3414864679 wait complete. 2024-04-25T19:03:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.336+0000 7f067a200640 1 Processor -- start 2024-04-25T19:03:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 -- start start 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741072a0 0x7f06741a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f067419a6c0 0x7f067419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0674112240 con 0x7f067419a6c0 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06741120c0 con 0x7f06741021d0 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.337+0000 7f067a200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f06741123c0 con 0x7f06741072a0 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41160/0 (socket says 172.21.15.63:41160) 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 -- 172.21.15.63:0/2063667041 learned_addr learned my addr 172.21.15.63:0/2063667041 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:26.339 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0672a00640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741072a0 0x7f06741a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673e00640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f067419a6c0 0x7f067419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 -- 172.21.15.63:0/2063667041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741072a0 msgr2=0x7f06741a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741072a0 0x7f06741a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 -- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f067419a6c0 msgr2=0x7f067419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f067419a6c0 0x7f067419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 -- 172.21.15.63:0/2063667041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067419b2c0 con 0x7f06741021d0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.338+0000 7f0673400640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f066400d950 tx=0x7f066400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.339+0000 7f066be00640 1 -- 172.21.15.63:0/2063667041 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664017030 con 0x7f06741021d0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.339+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06741a73a0 con 0x7f06741021d0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.339+0000 7f066be00640 1 -- 172.21.15.63:0/2063667041 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0664004020 con 0x7f06741021d0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.339+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a7800 con 0x7f06741021d0 2024-04-25T19:03:26.341 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.339+0000 7f066be00640 1 -- 172.21.15.63:0/2063667041 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06640047a0 con 0x7f06741021d0 2024-04-25T19:03:26.344 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.342+0000 7f067cfd3740 1 Processor -- start 2024-04-25T19:03:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 -- start start 2024-04-25T19:03:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f066be00640 1 -- 172.21.15.63:0/2063667041 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f06640041d0 con 0x7f06741021d0 2024-04-25T19:03:26.345 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc4424c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc444df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc39d900 con 0x55dfdc437820 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc343ef0 con 0x55dfdc442a00 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f067cfd3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc3998c0 con 0x55dfdc438140 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f0672a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc444df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f0673400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc4424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f0673e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54988/0 (socket says 172.21.15.63:54988) 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 -- 172.21.15.63:0/1460414307 learned_addr learned my addr 172.21.15.63:0/1460414307 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.343+0000 7f0672a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc444df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41164/0 (socket says 172.21.15.63:41164) 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 -- 172.21.15.63:0/1460414307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 msgr2=0x55dfdc4424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc4424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.346 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 -- 172.21.15.63:0/1460414307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 msgr2=0x55dfdc444df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc444df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673e00640 1 -- 172.21.15.63:0/1460414307 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfdc0b0210 con 0x55dfdc437820 2024-04-25T19:03:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.344+0000 7f0673400640 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc4424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f0673e00640 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f066c004af0 tx=0x7f066c009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57c2b89602f6e21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f065fe00640 1 -- 172.21.15.63:0/1460414307 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c031cf0 con 0x55dfdc437820 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f066be00640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 0x7f06580a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f065fe00640 1 -- 172.21.15.63:0/1460414307 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066c036040 con 0x55dfdc437820 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f065fe00640 1 -- 172.21.15.63:0/1460414307 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c0042a0 con 0x55dfdc437820 2024-04-25T19:03:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.345+0000 7f066be00640 1 -- 172.21.15.63:0/2063667041 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8587+0+0 (secure 0 0 0) 0x7f06640d0110 con 0x7f06741021d0 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f0673400640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 0x7f06580a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 -- 172.21.15.63:0/1460414307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 msgr2=0x55dfdc437c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f066c004af0 tx=0x7f066c009800 comp rx=0 tx=0).stop 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 -- 172.21.15.63:0/1460414307 shutdown_connections 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc444df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc4424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 --2- 172.21.15.63:0/1460414307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc437c00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f067cfd3740 1 -- 172.21.15.63:0/1460414307 >> 172.21.15.63:0/1460414307 conn(0x55dfdc433d30 msgr2=0x55dfdc434110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.346+0000 7f0673400640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 0x7f06580a6300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55dfdc438dc0 tx=0x7f06640d3bc0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.347+0000 7f067cfd3740 1 -- 172.21.15.63:0/1460414307 shutdown_connections 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.347+0000 7f067cfd3740 1 -- 172.21.15.63:0/1460414307 wait complete. 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.347+0000 7f067cfd3740 1 -- start start 2024-04-25T19:03:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.347+0000 7f067cfd3740 1 Processor -- start 2024-04-25T19:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc43f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc440420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc4565e0 con 0x55dfdc437820 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc357310 con 0x55dfdc442a00 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55dfdc456760 con 0x55dfdc438140 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f067cfd3740 10 client.0 _ll_register_callbacks cb 0x55dfdc325cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f0673e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc43f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f0673e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc43f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55000/0 (socket says 172.21.15.63:55000) 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f0673e00640 1 -- 172.21.15.63:0/479674507 learned_addr learned my addr 172.21.15.63:0/479674507 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.348+0000 7f0672a00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 -- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 msgr2=0x55dfdc440420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0673e00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc440420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc440420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 -- 172.21.15.63:0/479674507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 msgr2=0x55dfdc43f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc43f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 -- 172.21.15.63:0/479674507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfdc17c620 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0673e00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc440420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f0672a00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43fee0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f067419bf10 tx=0x7f0660012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34e7690935a4f2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.349+0000 7f065cc00640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660004030 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8587+0+0 (secure 0 0 0) 0x7f06600041d0 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f066c036040 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f06600043d0 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f067cfd3740 10 client.27145 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55dfdc462780 con 0x55dfdc438140 2024-04-25T19:03:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066c036040 con 0x55dfdc438140 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.350+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f06600039c0 con 0x55dfdc438140 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 10 client.27145 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 10 client.27145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 20 client.27145 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc463cb0 0x55dfdc466110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 4 client.27145 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfdc3b18b0 con 0x55dfdc463cb0 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.351+0000 7f0673e00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc463cb0 0x55dfdc466110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.352+0000 7f0673e00640 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc463cb0 0x55dfdc466110 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f250d322e76bbbcf server_cookie=be4a2ccef661b137 in_seq=0 out_seq=0 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.352+0000 7f065cc00640 10 client.27145 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:26.355 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.352+0000 7f065cc00640 1 -- 172.21.15.63:0/479674507 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dfdc3b18b0 con 0x55dfdc463cb0 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.352+0000 7f065cc00640 10 client.27145 handle_command_reply: tid=0 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 client.27145 shutdown 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 msgr2=0x55dfdc43fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43fee0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f067419bf10 tx=0x7f0660012b60 comp rx=0 tx=0).stop 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 shutdown_connections 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc463cb0 0x55dfdc466110 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc442a00 0x55dfdc440420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43fee0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.356 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 --2- 172.21.15.63:0/479674507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc437820 0x55dfdc43f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 >> 172.21.15.63:0/479674507 conn(0x55dfdc433d30 msgr2=0x55dfdc439a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f065cc00640 0 client.27145 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.353+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 shutdown_connections 2024-04-25T19:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.354+0000 7f067cfd3740 1 -- 172.21.15.63:0/479674507 wait complete. 2024-04-25T19:03:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.354+0000 7f067cfd3740 20 client.27145 trim_cache size 0 max 16384 2024-04-25T19:03:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.375+0000 7f067cfd3740 1 Processor -- start 2024-04-25T19:03:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.375+0000 7f067cfd3740 1 -- start start 2024-04-25T19:03:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 0x55dfdc45f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc46ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0654011210 con 0x55dfdc442a00 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f065400ea20 con 0x55dfdc437820 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f067cfd3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f065400c890 con 0x55dfdc438140 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f0672a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f0673e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 0x55dfdc45f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f0672a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:39664/0 (socket says 172.21.15.63:39664) 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f0672a00640 1 -- 172.21.15.63:0/2455997066 learned_addr learned my addr 172.21.15.63:0/2455997066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.376+0000 7f0673e00640 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc46ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 -- 172.21.15.63:0/2455997066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 msgr2=0x55dfdc43e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 -- 172.21.15.63:0/2455997066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 msgr2=0x55dfdc46ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc46ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 -- 172.21.15.63:0/2455997066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfdc437f70 con 0x55dfdc437820 2024-04-25T19:03:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0672a00640 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:26.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f0673e00640 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 0x55dfdc45f500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55dfdc466950 tx=0x7f066c03e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f7dfddff3812e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f065f400640 1 -- 172.21.15.63:0/2455997066 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c04a040 con 0x55dfdc437820 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f065f400640 1 -- 172.21.15.63:0/2455997066 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066c036040 con 0x55dfdc437820 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.377+0000 7f065f400640 1 -- 172.21.15.63:0/2455997066 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c02f870 con 0x55dfdc437820 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 -- 172.21.15.63:0/2455997066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 msgr2=0x55dfdc45f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 0x55dfdc45f500 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55dfdc466950 tx=0x7f066c03e040 comp rx=0 tx=0).stop 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 -- 172.21.15.63:0/2455997066 shutdown_connections 2024-04-25T19:03:26.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc46ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc438140 0x55dfdc43e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2455997066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc437820 0x55dfdc45f500 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.378+0000 7f067cfd3740 1 -- 172.21.15.63:0/2455997066 >> 172.21.15.63:0/2455997066 conn(0x55dfdc433d30 msgr2=0x55dfdc43e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.379+0000 7f067cfd3740 1 -- 172.21.15.63:0/2455997066 shutdown_connections 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.379+0000 7f067cfd3740 1 -- 172.21.15.63:0/2455997066 wait complete. 2024-04-25T19:03:26.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.379+0000 7f067cfd3740 1 -- start start 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.379+0000 7f067cfd3740 1 Processor -- start 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc437820 0x55dfdc4589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc459460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f065400cee0 con 0x55dfdc442a00 2024-04-25T19:03:26.388 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f065400e350 con 0x55dfdc438140 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f065400d5b0 con 0x55dfdc437820 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f067cfd3740 10 client.0 _ll_register_callbacks cb 0x7f0660009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f0672a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f0673e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc437820 0x55dfdc4589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f0672a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41192/0 (socket says 172.21.15.63:41192) 2024-04-25T19:03:26.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f0672a00640 1 -- 172.21.15.63:0/2892470057 learned_addr learned my addr 172.21.15.63:0/2892470057 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.380+0000 7f0673e00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc459460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 -- 172.21.15.63:0/2892470057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc437820 msgr2=0x55dfdc4589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc437820 0x55dfdc4589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 -- 172.21.15.63:0/2892470057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 msgr2=0x55dfdc459460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc459460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 -- 172.21.15.63:0/2892470057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfdc4c84a0 con 0x55dfdc438140 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f0672a00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0660012420 tx=0x7f066001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad758744c833e66a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f065cc00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660028040 con 0x55dfdc438140 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8587+0+0 (secure 0 0 0) 0x7f066000f8f0 con 0x55dfdc438140 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.381+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f066c036040 con 0x55dfdc438140 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.382+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f0660039050 con 0x55dfdc438140 2024-04-25T19:03:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.382+0000 7f067cfd3740 10 client.27137 fetch_fsmap learned FSMap version 50 2024-04-25T19:03:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.382+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x55dfdc4c97c0 con 0x55dfdc438140 2024-04-25T19:03:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f066c036040 con 0x55dfdc438140 2024-04-25T19:03:26.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 50) ==== 8354+0+0 (secure 0 0 0) 0x7f0660034030 con 0x55dfdc438140 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 10 client.27137 fetch_fsmap finished waiting for FSMap version 50 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 10 client.27137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 20 client.27137 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc4ca020 0x55dfdc4cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 4 client.27137 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:03:26.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.383+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dfdc3ab320 con 0x55dfdc4ca020 2024-04-25T19:03:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.384+0000 7f0673e00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc4ca020 0x55dfdc4cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.384+0000 7f065cc00640 10 client.27137 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:26.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.384+0000 7f0673e00640 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc4ca020 0x55dfdc4cc440 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74590f9f34bee232 server_cookie=2d79e92a078e9616 in_seq=0 out_seq=0 2024-04-25T19:03:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f065cc00640 1 -- 172.21.15.63:0/2892470057 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dfdc3ab320 con 0x55dfdc4ca020 2024-04-25T19:03:26.607 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f065cc00640 10 client.27137 handle_command_reply: tid=0 2024-04-25T19:03:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 client.27137 shutdown 2024-04-25T19:03:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 msgr2=0x55dfdc458f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.608 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0660012420 tx=0x7f066001c4b0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 shutdown_connections 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55dfdc4ca020 0x55dfdc4cc440 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55dfdc438140 0x55dfdc458f20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfdc442a00 0x55dfdc459460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 --2- 172.21.15.63:0/2892470057 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55dfdc437820 0x55dfdc4589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 >> 172.21.15.63:0/2892470057 conn(0x55dfdc433d30 msgr2=0x55dfdc43eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.606+0000 7f065cc00640 0 client.27137 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.607+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 shutdown_connections 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.607+0000 7f067cfd3740 1 -- 172.21.15.63:0/2892470057 wait complete. 2024-04-25T19:03:26.609 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.607+0000 7f067cfd3740 20 client.27137 trim_cache size 0 max 16384 2024-04-25T19:03:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:03:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:03:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:03:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 msgr2=0x7f06580a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 0x7f06580a6300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55dfdc438dc0 tx=0x7f06640d3bc0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 msgr2=0x7f06741a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f066400d950 tx=0x7f066400de10 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 shutdown_connections 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f06580a3e40 0x7f06580a6300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f067419a6c0 0x7f067419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f06741072a0 0x7f06741a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 --2- 172.21.15.63:0/2063667041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06741021d0 0x7f06741a0600 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 >> 172.21.15.63:0/2063667041 conn(0x7f06740793e0 msgr2=0x7f06740ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 shutdown_connections 2024-04-25T19:03:26.611 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:26.609+0000 7f067a200640 1 -- 172.21.15.63:0/2063667041 wait complete. 2024-04-25T19:03:26.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:26 smithi070 ceph-mon[32820]: pgmap v533: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:26 smithi114 ceph-mon[32988]: pgmap v533: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:26 smithi063 ceph-mon[28006]: pgmap v533: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:27.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 82729 19.48 MB/sec execute 85 sec latency 195.429 ms 2024-04-25T19:03:27.418 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setuser ceph since I am not root 2024-04-25T19:03:27.418 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setgroup ceph since I am not root 2024-04-25T19:03:27.418 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:27 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/2935281799 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c50dce3a40 con 0x55c50dcb7000 2024-04-25T19:03:27.418 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T19:03:27.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:27 smithi070 ceph-mon[32820]: daemon mds.f restarted 2024-04-25T19:03:27.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:27 smithi114 ceph-mon[32988]: daemon mds.f restarted 2024-04-25T19:03:27.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:27 smithi063 ceph-mon[28006]: daemon mds.f restarted 2024-04-25T19:03:28.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 83564 19.82 MB/sec execute 86 sec latency 49.031 ms 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: pgmap v534: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/380908850,v1:172.21.15.114:6835/380908850] up:boot 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:28 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: pgmap v534: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/380908850,v1:172.21.15.114:6835/380908850] up:boot 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:28 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: pgmap v534: 97 pgs: 97 active+clean; 748 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.0 MiB/s wr, 520 op/s 2024-04-25T19:03:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: osdmap e122: 12 total, 12 up, 12 in 2024-04-25T19:03:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/380908850,v1:172.21.15.114:6835/380908850] up:boot 2024-04-25T19:03:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:28 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:29.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 84298 20.14 MB/sec execute 87 sec latency 47.160 ms 2024-04-25T19:03:30.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 84846 20.25 MB/sec execute 88 sec latency 63.859 ms 2024-04-25T19:03:30.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:30 smithi070 ceph-mon[32820]: pgmap v536: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:30 smithi114 ceph-mon[32988]: pgmap v536: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:30.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:30 smithi063 ceph-mon[28006]: pgmap v536: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:31.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 85498 20.39 MB/sec execute 89 sec latency 57.815 ms 2024-04-25T19:03:32.096 INFO:tasks.workunit.client.0.smithi063.stdout: 10 86130 20.48 MB/sec execute 90 sec latency 107.771 ms 2024-04-25T19:03:32.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:32 smithi070 ceph-mon[32820]: pgmap v537: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:32 smithi114 ceph-mon[32988]: pgmap v537: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:32.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:32 smithi063 ceph-mon[28006]: pgmap v537: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:33.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 86898 20.74 MB/sec execute 91 sec latency 46.759 ms 2024-04-25T19:03:34.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 87663 20.96 MB/sec execute 92 sec latency 39.104 ms 2024-04-25T19:03:34.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:34 smithi070 ceph-mon[32820]: pgmap v538: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:34 smithi114 ceph-mon[32988]: pgmap v538: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:34.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:34 smithi063 ceph-mon[28006]: pgmap v538: 97 pgs: 97 active+clean; 759 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 621 op/s 2024-04-25T19:03:35.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 88233 21.06 MB/sec execute 93 sec latency 43.587 ms 2024-04-25T19:03:36.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 88882 21.20 MB/sec execute 94 sec latency 61.173 ms 2024-04-25T19:03:36.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:36 smithi070 ceph-mon[32820]: pgmap v539: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:36.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:36 smithi114 ceph-mon[32988]: pgmap v539: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:36 smithi063 ceph-mon[28006]: pgmap v539: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:37.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 89694 21.43 MB/sec execute 95 sec latency 53.084 ms 2024-04-25T19:03:38.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 90418 21.70 MB/sec execute 96 sec latency 41.290 ms 2024-04-25T19:03:38.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:38 smithi070 ceph-mon[32820]: pgmap v540: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:38.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:38 smithi114 ceph-mon[32988]: pgmap v540: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:38.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:38 smithi063 ceph-mon[28006]: pgmap v540: 97 pgs: 97 active+clean; 787 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.8 MiB/s wr, 612 op/s 2024-04-25T19:03:39.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 91024 21.80 MB/sec execute 97 sec latency 44.106 ms 2024-04-25T19:03:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.801+0000 7f69f8e00640 1 -- 172.21.15.63:0/1544895778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f410c330 msgr2=0x7f69f410e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.801+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1544895778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f410c330 0x7f69f410e7d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009ca0 tx=0x7f69ec02a950 comp rx=0 tx=0).stop 2024-04-25T19:03:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 -- 172.21.15.63:0/1544895778 shutdown_connections 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1544895778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4103830 0x7f69f410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1544895778 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f410c330 0x7f69f410e7d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1544895778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f4102f10 0x7f69f41032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 -- 172.21.15.63:0/1544895778 >> 172.21.15.63:0/1544895778 conn(0x7f69f40fc770 msgr2=0x7f69f40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 -- 172.21.15.63:0/1544895778 shutdown_connections 2024-04-25T19:03:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.802+0000 7f69f8e00640 1 -- 172.21.15.63:0/1544895778 wait complete. 2024-04-25T19:03:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.803+0000 7f69f8e00640 1 Processor -- start 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.803+0000 7f69f8e00640 1 -- start start 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.803+0000 7f69f8e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 0x7f69f4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f8e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 0x7f69f4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f8e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 0x7f69f419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f8e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4111fc0 con 0x7f69f410c330 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f8e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4111e40 con 0x7f69f4102f10 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f8e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4112140 con 0x7f69f4103830 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 0x7f69f4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 0x7f69f4198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39670/0 (socket says 172.21.15.63:39670) 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 0x7f69f4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f3400640 1 -- 172.21.15.63:0/1775154134 learned_addr learned my addr 172.21.15.63:0/1775154134 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f3e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 0x7f69f419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 -- 172.21.15.63:0/1775154134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 msgr2=0x7f69f4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 0x7f69f4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 -- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 msgr2=0x7f69f419ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 0x7f69f419ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f2a00640 1 -- 172.21.15.63:0/1775154134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f419d530 con 0x7f69f4103830 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.804+0000 7f69f3e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 0x7f69f419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69f2a00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 0x7f69f4198a80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e7c0 tx=0x7f69e000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69ebe00640 1 -- 172.21.15.63:0/1775154134 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e0009530 con 0x7f69f4103830 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f419d820 con 0x7f69f4103830 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69ebe00640 1 -- 172.21.15.63:0/1775154134 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e000ccc0 con 0x7f69f4103830 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69ebe00640 1 -- 172.21.15.63:0/1775154134 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e001c070 con 0x7f69f4103830 2024-04-25T19:03:39.808 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.805+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f41a50b0 con 0x7f69f4103830 2024-04-25T19:03:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.809+0000 7f69fd1ad740 1 Processor -- start 2024-04-25T19:03:39.810 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.809+0000 7f69fd1ad740 1 -- start start 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c02ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55971bf87900 con 0x55971c02cb90 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55971bf2def0 con 0x55971c0222d0 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55971bf838c0 con 0x55971c0219b0 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54228/0 (socket says 172.21.15.63:54228) 2024-04-25T19:03:39.812 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 -- 172.21.15.63:0/657733044 learned_addr learned my addr 172.21.15.63:0/657733044 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69ebe00640 1 -- 172.21.15.63:0/1775154134 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f69e00049a0 con 0x7f69f4103830 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 -- 172.21.15.63:0/657733044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 msgr2=0x55971c02c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c02c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.810+0000 7f69f3e00640 1 -- 172.21.15.63:0/657733044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 msgr2=0x55971c02ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.811+0000 7f69f3e00640 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c02ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.813 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.811+0000 7f69f3e00640 1 -- 172.21.15.63:0/657733044 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971bc9a210 con 0x55971c0219b0 2024-04-25T19:03:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.811+0000 7f69f3e00640 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009ca0 tx=0x7f69ec009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3577e605cf952ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.811+0000 7f69e8c00640 1 -- 172.21.15.63:0/657733044 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec048070 con 0x55971c0219b0 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.812+0000 7f69e8c00640 1 -- 172.21.15.63:0/657733044 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69ec003e20 con 0x55971c0219b0 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.812+0000 7f69e8c00640 1 -- 172.21.15.63:0/657733044 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec034040 con 0x55971c0219b0 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.812+0000 7f69ebe00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 0x7f69d80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.812+0000 7f69ebe00640 1 -- 172.21.15.63:0/1775154134 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8543+0+0 (secure 0 0 0) 0x7f69e00d0670 con 0x7f69f4103830 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 -- 172.21.15.63:0/657733044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 msgr2=0x55971c021d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009ca0 tx=0x7f69ec009360 comp rx=0 tx=0).stop 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69f3400640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 0x7f69d80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 -- 172.21.15.63:0/657733044 shutdown_connections 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c02c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c02ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/657733044 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c021d90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 -- 172.21.15.63:0/657733044 >> 172.21.15.63:0/657733044 conn(0x55971c01dec0 msgr2=0x55971c01e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 -- 172.21.15.63:0/657733044 shutdown_connections 2024-04-25T19:03:39.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.813+0000 7f69fd1ad740 1 -- 172.21.15.63:0/657733044 wait complete. 2024-04-25T19:03:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.814+0000 7f69f3400640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 0x7f69d80a61d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55971c022f50 tx=0x7f69e4005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.818 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.814+0000 7f69fd1ad740 1 -- start start 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.814+0000 7f69fd1ad740 1 Processor -- start 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c025d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 0x55971c026280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55971c02f6a0 con 0x55971c0222d0 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55971bf41310 con 0x55971c02cb90 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55971c02f820 con 0x55971c0219b0 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69fd1ad740 10 client.0 _ll_register_callbacks cb 0x55971bf0fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c025d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39688/0 (socket says 172.21.15.63:39688) 2024-04-25T19:03:39.819 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 -- 172.21.15.63:0/2894454534 learned_addr learned my addr 172.21.15.63:0/2894454534 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f2a00640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 0x55971c026280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 -- 172.21.15.63:0/2894454534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 msgr2=0x55971c025d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c025d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 -- 172.21.15.63:0/2894454534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 msgr2=0x55971c026280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.815+0000 7f69f3e00640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 0x55971c026280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69f3e00640 1 -- 172.21.15.63:0/2894454534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971bd66620 con 0x55971c02cb90 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69f2a00640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 0x55971c026280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69f3e00640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f69ec005cc0 tx=0x7f69ec02f650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5ac13e231f364e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69dd600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec051060 con 0x55971c02cb90 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69ec003e20 con 0x55971c02cb90 2024-04-25T19:03:39.820 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8543+0+0 (secure 0 0 0) 0x7f69ec040050 con 0x55971c02cb90 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f69ec048030 con 0x55971c02cb90 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69fd1ad740 10 client.27155 fetch_fsmap learned FSMap version 52 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.816+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55971c0442c0 con 0x55971c02cb90 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69ec003e20 con 0x55971c02cb90 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f69ec0305c0 con 0x55971c02cb90 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 10 client.27155 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 10 client.27155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 20 client.27155 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c045610 0x55971c047a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 4 client.27155 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55971bf9b8b0 con 0x55971c045610 2024-04-25T19:03:39.821 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.817+0000 7f69f3400640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c045610 0x55971c047a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.818+0000 7f69f3400640 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c045610 0x55971c047a00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e9a27bddae7709e server_cookie=65193a04a1fcc2bf in_seq=0 out_seq=0 2024-04-25T19:03:39.822 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.818+0000 7f69dd600640 10 client.27155 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:39.823 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69dd600640 1 -- 172.21.15.63:0/2894454534 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55971bf9b8b0 con 0x55971c045610 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69dd600640 10 client.27155 handle_command_reply: tid=0 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 client.27155 shutdown 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 msgr2=0x55971c02a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f69ec005cc0 tx=0x7f69ec02f650 comp rx=0 tx=0).stop 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 shutdown_connections 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c045610 0x55971c047a00 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c02cb90 0x55971c02a5a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0222d0 0x55971c026280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/2894454534 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0219b0 0x55971c025d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69dd600640 0 client.27155 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 >> 172.21.15.63:0/2894454534 conn(0x55971c01dec0 msgr2=0x55971c023bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 shutdown_connections 2024-04-25T19:03:39.824 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 1 -- 172.21.15.63:0/2894454534 wait complete. 2024-04-25T19:03:39.825 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.822+0000 7f69fd1ad740 20 client.27155 trim_cache size 0 max 16384 2024-04-25T19:03:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 Processor -- start 2024-04-25T19:03:39.844 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 -- start start 2024-04-25T19:03:39.845 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 0x55971c01cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c04d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41ae0a0 con 0x55971c02cb90 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41ab8b0 con 0x55971c0219b0 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.843+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41a9700 con 0x55971c0222d0 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f2a00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 0x55971c01cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39698/0 (socket says 172.21.15.63:39698) 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 -- 172.21.15.63:0/399199037 learned_addr learned my addr 172.21.15.63:0/399199037 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3e00640 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c04d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.846 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 -- 172.21.15.63:0/399199037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 msgr2=0x55971c01cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 0x55971c01cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 -- 172.21.15.63:0/399199037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 msgr2=0x55971c04d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c04d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 -- 172.21.15.63:0/399199037 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971c022a30 con 0x55971c0219b0 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f2a00640 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 0x55971c01cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:03:39.847 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.844+0000 7f69f3400640 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f69e4013010 tx=0x7f69e401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4303a0593a1b3010 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.845+0000 7f69dfe00640 1 -- 172.21.15.63:0/399199037 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e401fd10 con 0x55971c0219b0 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.845+0000 7f69dfe00640 1 -- 172.21.15.63:0/399199037 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e402b920 con 0x55971c0219b0 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.845+0000 7f69dfe00640 1 -- 172.21.15.63:0/399199037 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e40047a0 con 0x55971c0219b0 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 -- 172.21.15.63:0/399199037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 msgr2=0x55971c02a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f69e4013010 tx=0x7f69e401b000 comp rx=0 tx=0).stop 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 -- 172.21.15.63:0/399199037 shutdown_connections 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c02cb90 0x55971c04d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.849 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c0222d0 0x55971c01cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/399199037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0219b0 0x55971c02a590 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 -- 172.21.15.63:0/399199037 >> 172.21.15.63:0/399199037 conn(0x55971c01dec0 msgr2=0x55971c01c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 -- 172.21.15.63:0/399199037 shutdown_connections 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.846+0000 7f69fd1ad740 1 -- 172.21.15.63:0/399199037 wait complete. 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 -- start start 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 Processor -- start 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0219b0 0x55971c041af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c02cb90 0x55971c03f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41a9d70 con 0x55971c0219b0 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41ab1e0 con 0x55971c0222d0 2024-04-25T19:03:39.850 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.847+0000 7f69fd1ad740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f69f41aa440 con 0x55971c02cb90 2024-04-25T19:03:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:39700/0 (socket says 172.21.15.63:39700) 2024-04-25T19:03:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0219b0 0x55971c041af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.851 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 -- 172.21.15.63:0/942101214 learned_addr learned my addr 172.21.15.63:0/942101214 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 -- 172.21.15.63:0/942101214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c02cb90 msgr2=0x55971c03f080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:03:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c02cb90 0x55971c03f080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.852 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 -- 172.21.15.63:0/942101214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0219b0 msgr2=0x55971c041af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0219b0 0x55971c041af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 -- 172.21.15.63:0/942101214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55971c02f1b0 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69fd1ad740 10 client.0 _ll_register_callbacks cb 0x7f69ec009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55971bf95320 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69f2a00640 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f69e000c6a0 tx=0x7f69e009c1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5303e222fbd8a2de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.848+0000 7f69dd600640 10 client.27161 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69dd600640 1 -- 172.21.15.63:0/942101214 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e00ce1d0 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69dd600640 1 -- 172.21.15.63:0/942101214 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 8543+0+0 (secure 0 0 0) 0x7f69e00bb2b0 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69dd600640 1 -- 172.21.15.63:0/942101214 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f69e00bb4f0 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 10 client.27161 fetch_fsmap learned FSMap version 52 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=52}) -- 0x55971c0ab210 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69dd600640 1 -- 172.21.15.63:0/942101214 <== mon.2 v2:172.21.15.70:3300/0 4 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f69e001f030 con 0x55971c0222d0 2024-04-25T19:03:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 10 client.27161 fetch_fsmap finished waiting for FSMap version 52 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 10 client.27161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 20 client.27161 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c0ab8b0 0x55971c0adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 4 client.27161 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.849+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55971bf8f190 con 0x55971c0ab8b0 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.850+0000 7f69f3400640 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c0ab8b0 0x55971c0adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.850+0000 7f69f3400640 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c0ab8b0 0x55971c0adcd0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f17ff30dc86035e7 server_cookie=8cc3e2066d4b0550 in_seq=0 out_seq=0 2024-04-25T19:03:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:39.850+0000 7f69dd600640 10 client.27161 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:40.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 91492 21.80 MB/sec execute 98 sec latency 197.797 ms 2024-04-25T19:03:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69dd600640 1 -- 172.21.15.63:0/942101214 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55971bf8f190 con 0x55971c0ab8b0 2024-04-25T19:03:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69dd600640 10 client.27161 handle_command_reply: tid=0 2024-04-25T19:03:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69fd1ad740 1 client.27161 shutdown 2024-04-25T19:03:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 msgr2=0x55971c042030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f69e000c6a0 tx=0x7f69e009c1a0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 shutdown_connections 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.107+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55971c0ab8b0 0x55971c0adcd0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55971c0222d0 0x55971c042030 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55971c02cb90 0x55971c03f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 --2- 172.21.15.63:0/942101214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55971c0219b0 0x55971c041af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 >> 172.21.15.63:0/942101214 conn(0x55971c01dec0 msgr2=0x55971c023360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69dd600640 0 client.27161 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 shutdown_connections 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 1 -- 172.21.15.63:0/942101214 wait complete. 2024-04-25T19:03:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.108+0000 7f69fd1ad740 20 client.27161 trim_cache size 0 max 16384 2024-04-25T19:03:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:03:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:03:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:03:40.111 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 msgr2=0x7f69d80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 0x7f69d80a61d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x55971c022f50 tx=0x7f69e4005f90 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 msgr2=0x7f69f4198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 0x7f69f4198a80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f69e000e7c0 tx=0x7f69e000ec80 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 shutdown_connections 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f69d80a3d10 0x7f69d80a61d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69f410c330 0x7f69f419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f69f4103830 0x7f69f4198a80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 --2- 172.21.15.63:0/1775154134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69f4102f10 0x7f69f4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 >> 172.21.15.63:0/1775154134 conn(0x7f69f40fc770 msgr2=0x7f69f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.109+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 shutdown_connections 2024-04-25T19:03:40.112 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:40.110+0000 7f69f8e00640 1 -- 172.21.15.63:0/1775154134 wait complete. 2024-04-25T19:03:40.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setuser ceph since I am not root 2024-04-25T19:03:40.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setgroup ceph since I am not root 2024-04-25T19:03:40.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/1729840614 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d92f975a40 con 0x55d92f948c00 2024-04-25T19:03:40.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T19:03:40.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-mon[32988]: pgmap v541: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 719 op/s 2024-04-25T19:03:40.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:40 smithi114 ceph-mon[32988]: daemon mds.f restarted 2024-04-25T19:03:40.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:40 smithi070 ceph-mon[32820]: pgmap v541: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 719 op/s 2024-04-25T19:03:40.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:40 smithi070 ceph-mon[32820]: daemon mds.f restarted 2024-04-25T19:03:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:40 smithi063 ceph-mon[28006]: pgmap v541: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.4 MiB/s wr, 719 op/s 2024-04-25T19:03:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:40 smithi063 ceph-mon[28006]: daemon mds.f restarted 2024-04-25T19:03:41.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 92095 21.89 MB/sec execute 99 sec latency 213.892 ms 2024-04-25T19:03:41.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:41 smithi070 ceph-mon[32820]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T19:03:41.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:41 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/198947212,v1:172.21.15.114:6835/198947212] up:boot 2024-04-25T19:03:41.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:41 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:41.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:41 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:41.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:41 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:41 smithi114 ceph-mon[32988]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T19:03:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:41 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/198947212,v1:172.21.15.114:6835/198947212] up:boot 2024-04-25T19:03:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:41 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:41 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:41.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:41 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:41.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:41 smithi063 ceph-mon[28006]: osdmap e123: 12 total, 12 up, 12 in 2024-04-25T19:03:41.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:41 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/198947212,v1:172.21.15.114:6835/198947212] up:boot 2024-04-25T19:03:41.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:41 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-04-25T19:03:41.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:41 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:03:41.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:41 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:03:42.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 92788 21.96 MB/sec execute 100 sec latency 155.595 ms 2024-04-25T19:03:42.718 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:42 smithi114 ceph-mon[32988]: pgmap v543: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:42 smithi063 ceph-mon[28006]: pgmap v543: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:42 smithi070 ceph-mon[32820]: pgmap v543: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:43.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 93639 22.32 MB/sec execute 101 sec latency 49.518 ms 2024-04-25T19:03:44.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 94253 22.47 MB/sec execute 102 sec latency 61.473 ms 2024-04-25T19:03:44.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:44 smithi114 ceph-mon[32988]: pgmap v544: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:44.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:44 smithi063 ceph-mon[28006]: pgmap v544: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:44 smithi070 ceph-mon[32820]: pgmap v544: 97 pgs: 97 active+clean; 815 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 492 op/s 2024-04-25T19:03:45.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 94805 22.48 MB/sec execute 103 sec latency 55.169 ms 2024-04-25T19:03:46.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 95210 22.49 MB/sec execute 104 sec latency 196.192 ms 2024-04-25T19:03:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:46 smithi114 ceph-mon[32988]: pgmap v545: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:46 smithi063 ceph-mon[28006]: pgmap v545: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:46 smithi070 ceph-mon[32820]: pgmap v545: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:47.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 95911 22.56 MB/sec execute 105 sec latency 48.084 ms 2024-04-25T19:03:48.097 INFO:tasks.workunit.client.0.smithi063.stdout: 10 96661 22.74 MB/sec execute 106 sec latency 48.825 ms 2024-04-25T19:03:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:48 smithi114 ceph-mon[32988]: pgmap v546: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:48.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:48 smithi063 ceph-mon[28006]: pgmap v546: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:48.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:48 smithi070 ceph-mon[32820]: pgmap v546: 97 pgs: 97 active+clean; 824 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.8 MiB/s wr, 626 op/s 2024-04-25T19:03:49.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 97312 22.90 MB/sec execute 107 sec latency 49.017 ms 2024-04-25T19:03:50.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 97945 22.96 MB/sec execute 108 sec latency 70.025 ms 2024-04-25T19:03:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:50 smithi114 ceph-mon[32988]: pgmap v547: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 704 op/s 2024-04-25T19:03:50.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:50 smithi063 ceph-mon[28006]: pgmap v547: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 704 op/s 2024-04-25T19:03:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:50 smithi070 ceph-mon[32820]: pgmap v547: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 10 MiB/s wr, 704 op/s 2024-04-25T19:03:51.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 98372 23.02 MB/sec execute 109 sec latency 320.412 ms 2024-04-25T19:03:52.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 99084 23.10 MB/sec execute 110 sec latency 98.362 ms 2024-04-25T19:03:52.755 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:52 smithi070 ceph-mon[32820]: pgmap v548: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 645 op/s 2024-04-25T19:03:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:52 smithi114 ceph-mon[32988]: pgmap v548: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 645 op/s 2024-04-25T19:03:52.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:52 smithi063 ceph-mon[28006]: pgmap v548: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.2 MiB/s wr, 645 op/s 2024-04-25T19:03:53.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 99749 23.26 MB/sec execute 111 sec latency 51.194 ms 2024-04-25T19:03:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 -- 172.21.15.63:0/3436433831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 msgr2=0x7f63981047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 --2- 172.21.15.63:0/3436433831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981047a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6388009ca0 tx=0x7f638802a8e0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 -- 172.21.15.63:0/3436433831 shutdown_connections 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 --2- 172.21.15.63:0/3436433831 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6398104d70 0x7f6398108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 --2- 172.21.15.63:0/3436433831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f6398111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 --2- 172.21.15.63:0/3436433831 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981047a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 -- 172.21.15.63:0/3436433831 >> 172.21.15.63:0/3436433831 conn(0x7f63980ffc20 msgr2=0x7f6398102040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 -- 172.21.15.63:0/3436433831 shutdown_connections 2024-04-25T19:03:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.282+0000 7f639ec00640 1 -- 172.21.15.63:0/3436433831 wait complete. 2024-04-25T19:03:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.283+0000 7f639ec00640 1 Processor -- start 2024-04-25T19:03:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.283+0000 7f639ec00640 1 -- start start 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f63981a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 0x7f639819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6398114450 con 0x7f6398109490 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63981142d0 con 0x7f639819a770 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639ec00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63981145d0 con 0x7f63981043c0 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56926/0 (socket says 172.21.15.63:56926) 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f6397e00640 1 -- 172.21.15.63:0/3069719007 learned_addr learned my addr 172.21.15.63:0/3069719007 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 0x7f639819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.284+0000 7f6397400640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f63981a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 -- 172.21.15.63:0/3069719007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 msgr2=0x7f63981a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 -- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 msgr2=0x7f63981a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f63981a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 -- 172.21.15.63:0/3069719007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639819b370 con 0x7f639819a770 2024-04-25T19:03:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f6397400640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f63981a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.285+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 0x7f639819abf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f639000e970 tx=0x7f639000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.286+0000 7f6394c00640 1 -- 172.21.15.63:0/3069719007 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000ccc0 con 0x7f639819a770 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.286+0000 7f6394c00640 1 -- 172.21.15.63:0/3069719007 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6390005020 con 0x7f639819a770 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.286+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63981a7450 con 0x7f639819a770 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.286+0000 7f6394c00640 1 -- 172.21.15.63:0/3069719007 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390004a00 con 0x7f639819a770 2024-04-25T19:03:53.289 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.286+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63981a7850 con 0x7f639819a770 2024-04-25T19:03:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.290+0000 7f6394c00640 1 -- 172.21.15.63:0/3069719007 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f6390004ba0 con 0x7f639819a770 2024-04-25T19:03:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.290+0000 7f63a1a1f740 1 Processor -- start 2024-04-25T19:03:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 -- start start 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd532900 con 0x55f6dd5d7a00 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd4d8ef0 con 0x55f6dd5cc820 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd52e8c0 con 0x55f6dd5cd140 2024-04-25T19:03:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f6397400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56932/0 (socket says 172.21.15.63:56932) 2024-04-25T19:03:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.291+0000 7f6397e00640 1 -- 172.21.15.63:0/154810142 learned_addr learned my addr 172.21.15.63:0/154810142 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6394c00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 0x7f637c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397400640 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6394c00640 1 -- 172.21.15.63:0/3069719007 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8543+0+0 (secure 0 0 0) 0x7f63900d07d0 con 0x7f639819a770 2024-04-25T19:03:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 -- 172.21.15.63:0/154810142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 msgr2=0x55f6dd5ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 -- 172.21.15.63:0/154810142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5d7a00 msgr2=0x55f6dd5d9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 -- 172.21.15.63:0/154810142 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6dd245210 con 0x55f6dd5cd140 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.292+0000 7f6397e00640 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6388031520 tx=0x7f6388031550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20a5331cf8e90085 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.293+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 0x7f637c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.293+0000 7f638d200640 1 -- 172.21.15.63:0/154810142 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388031ad0 con 0x55f6dd5cd140 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.293+0000 7f638d200640 1 -- 172.21.15.63:0/154810142 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6388036030 con 0x55f6dd5cd140 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.293+0000 7f638d200640 1 -- 172.21.15.63:0/154810142 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388004680 con 0x55f6dd5cd140 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f639cc00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 0x7f637c0a6380 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63900a4e40 tx=0x7f6390005460 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- 172.21.15.63:0/154810142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 msgr2=0x55f6dd5d74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6388031520 tx=0x7f6388031550 comp rx=0 tx=0).stop 2024-04-25T19:03:53.297 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- 172.21.15.63:0/154810142 shutdown_connections 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d74c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/154810142 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- 172.21.15.63:0/154810142 >> 172.21.15.63:0/154810142 conn(0x55f6dd5c8d30 msgr2=0x55f6dd5c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- 172.21.15.63:0/154810142 shutdown_connections 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- 172.21.15.63:0/154810142 wait complete. 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.294+0000 7f63a1a1f740 1 -- start start 2024-04-25T19:03:53.298 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 Processor -- start 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd5da5d0 con 0x55f6dd5cc820 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd4ec310 con 0x55f6dd5d7a00 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f6dd5da750 con 0x55f6dd5cd140 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.295+0000 7f63a1a1f740 10 client.0 _ll_register_callbacks cb 0x55f6dd4bacb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.302 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51586/0 (socket says 172.21.15.63:51586) 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397400640 1 -- 172.21.15.63:0/4041178746 learned_addr learned my addr 172.21.15.63:0/4041178746 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397400640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 -- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 msgr2=0x55f6dd5d53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 -- 172.21.15.63:0/4041178746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 msgr2=0x55f6dd5d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.296+0000 7f6397e00640 1 -- 172.21.15.63:0/4041178746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6dd311620 con 0x55f6dd5cd140 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f6397e00640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d4e90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6388004af0 tx=0x7f6388004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26df0c035fc8b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f6381600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388049040 con 0x55f6dd5cd140 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6388036030 con 0x55f6dd5cd140 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8543+0+0 (secure 0 0 0) 0x7f6388003b70 con 0x55f6dd5cd140 2024-04-25T19:03:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f638805d050 con 0x55f6dd5cd140 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.297+0000 7f63a1a1f740 10 client.27190 fetch_fsmap learned FSMap version 54 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.298+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55f6dd5da5d0 con 0x55f6dd5cd140 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.298+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6388036030 con 0x55f6dd5cd140 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.298+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f638804c030 con 0x55f6dd5cd140 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.298+0000 7f63a1a1f740 10 client.27190 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.298+0000 7f63a1a1f740 10 client.27190 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.299+0000 7f63a1a1f740 20 client.27190 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.299+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd5de940 0x55f6dd5e0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.299+0000 7f63a1a1f740 4 client.27190 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.299+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6dd5468b0 con 0x55f6dd5de940 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.299+0000 7f6397400640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd5de940 0x55f6dd5e0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.300+0000 7f6397400640 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd5de940 0x55f6dd5e0de0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9477898a161f278 server_cookie=2d6c3a240309ec4c in_seq=0 out_seq=0 2024-04-25T19:03:53.304 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.300+0000 7f6381600640 10 client.27190 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.301+0000 7f6381600640 1 -- 172.21.15.63:0/4041178746 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f6dd5468b0 con 0x55f6dd5de940 2024-04-25T19:03:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.301+0000 7f6381600640 10 client.27190 handle_command_reply: tid=0 2024-04-25T19:03:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.301+0000 7f63a1a1f740 1 client.27190 shutdown 2024-04-25T19:03:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 msgr2=0x55f6dd5d4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.305 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d4e90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6388004af0 tx=0x7f6388004990 comp rx=0 tx=0).stop 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 shutdown_connections 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd5de940 0x55f6dd5e0de0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5d53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5d4e90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/4041178746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 >> 172.21.15.63:0/4041178746 conn(0x55f6dd5c8d30 msgr2=0x55f6dd5cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f6381600640 0 client.27190 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 shutdown_connections 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 1 -- 172.21.15.63:0/4041178746 wait complete. 2024-04-25T19:03:53.306 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.302+0000 7f63a1a1f740 20 client.27190 trim_cache size 0 max 16384 2024-04-25T19:03:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 Processor -- start 2024-04-25T19:03:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 -- start start 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cd140 0x55f6dd5ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5d7a00 0x55f6dd602290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f63981ae210 con 0x55f6dd5cd140 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63981aba20 con 0x55f6dd5cc820 2024-04-25T19:03:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63981a9870 con 0x55f6dd5d7a00 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f6397400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f6397400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:60346/0 (socket says 172.21.15.63:60346) 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f6397400640 1 -- 172.21.15.63:0/1148376946 learned_addr learned my addr 172.21.15.63:0/1148376946 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f6397e00640 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cd140 0x55f6dd5ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.322+0000 7f6397400640 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5d7a00 0x55f6dd602290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 -- 172.21.15.63:0/1148376946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5d7a00 msgr2=0x55f6dd602290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5d7a00 0x55f6dd602290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 -- 172.21.15.63:0/1148376946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cd140 msgr2=0x55f6dd5ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cd140 0x55f6dd5ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 -- 172.21.15.63:0/1148376946 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6dd5c4690 con 0x55f6dd5cc820 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.323+0000 7f6397400640 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f639806b660 tx=0x7f63840046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2398e68d082d7392 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.324+0000 7f6383e00640 1 -- 172.21.15.63:0/1148376946 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384017cc0 con 0x55f6dd5cc820 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.324+0000 7f6383e00640 1 -- 172.21.15.63:0/1148376946 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6384017e60 con 0x55f6dd5cc820 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.324+0000 7f6383e00640 1 -- 172.21.15.63:0/1148376946 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384025350 con 0x55f6dd5cc820 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.324+0000 7f63a1a1f740 1 -- 172.21.15.63:0/1148376946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 msgr2=0x55f6dd5d3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.324+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f639806b660 tx=0x7f63840046e0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 -- 172.21.15.63:0/1148376946 shutdown_connections 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5d7a00 0x55f6dd602290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cd140 0x55f6dd5ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/1148376946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5cc820 0x55f6dd5d3ab0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 -- 172.21.15.63:0/1148376946 >> 172.21.15.63:0/1148376946 conn(0x55f6dd5c8d30 msgr2=0x55f6dd5ce1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 -- 172.21.15.63:0/1148376946 shutdown_connections 2024-04-25T19:03:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 -- 172.21.15.63:0/1148376946 wait complete. 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 -- start start 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.325+0000 7f63a1a1f740 1 Processor -- start 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5e40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f63981a9ee0 con 0x55f6dd5cc820 2024-04-25T19:03:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63981ab350 con 0x55f6dd5d7a00 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f63981aa5b0 con 0x55f6dd5cd140 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f63a1a1f740 10 client.0 _ll_register_callbacks cb 0x7f6388004850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f6397400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f6397e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:56962/0 (socket says 172.21.15.63:56962) 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f6397e00640 1 -- 172.21.15.63:0/3758195905 learned_addr learned my addr 172.21.15.63:0/3758195905 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.326+0000 7f6397400640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5e40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 -- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 msgr2=0x55f6dd5e40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5e40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 -- 172.21.15.63:0/3758195905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 msgr2=0x55f6dd5e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 -- 172.21.15.63:0/3758195905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6dd5e48b0 con 0x55f6dd5cd140 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6397e00640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6388002230 tx=0x7f6388031d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d13c08fc341222b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:03:53.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6381600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388003b70 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8543+0+0 (secure 0 0 0) 0x7f63880305e0 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.327+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6384017e60 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.328+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7f6388009540 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.328+0000 7f63a1a1f740 10 client.27196 fetch_fsmap learned FSMap version 54 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.328+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) -- 0x55f6dd65eab0 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.328+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6384017e60 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.328+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 54) ==== 8354+0+0 (secure 0 0 0) 0x7f638804c040 con 0x55f6dd5cd140 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 10 client.27196 fetch_fsmap finished waiting for FSMap version 54 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 10 client.27196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 20 client.27196 populate_metadata read hostname 'smithi063' 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd65f290 0x55f6dd6616b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 4 client.27196 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6dd540320 con 0x55f6dd65f290 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f6397400640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd65f290 0x55f6dd6616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:03:53.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f6397400640 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd65f290 0x55f6dd6616b0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ae86560730e6f1e server_cookie=7bce1d3742899fc8 in_seq=0 out_seq=0 2024-04-25T19:03:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.329+0000 7f6381600640 10 client.27196 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.561+0000 7f6381600640 1 -- 172.21.15.63:0/3758195905 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6dd540320 con 0x55f6dd65f290 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.561+0000 7f6381600640 10 client.27196 handle_command_reply: tid=0 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.561+0000 7f63a1a1f740 1 client.27196 shutdown 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.561+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 msgr2=0x55f6dd5e3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.561+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f6388002230 tx=0x7f6388031d90 comp rx=0 tx=0).stop 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 shutdown_connections 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f6dd65f290 0x55f6dd6616b0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f6dd5d7a00 0x55f6dd5e40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f6dd5cd140 0x55f6dd5e3b70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 --2- 172.21.15.63:0/3758195905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f6dd5cc820 0x55f6dd5e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f6381600640 0 client.27196 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:03:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 >> 172.21.15.63:0/3758195905 conn(0x55f6dd5c8d30 msgr2=0x55f6dd5cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 shutdown_connections 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 1 -- 172.21.15.63:0/3758195905 wait complete. 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.562+0000 7f63a1a1f740 20 client.27196 trim_cache size 0 max 16384 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:03:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 msgr2=0x7f637c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 0x7f637c0a6380 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63900a4e40 tx=0x7f6390005460 comp rx=0 tx=0).stop 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 msgr2=0x7f639819abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 0x7f639819abf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f639000e970 tx=0x7f639000ee30 comp rx=0 tx=0).stop 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 shutdown_connections 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f637c0a3ec0 0x7f637c0a6380 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.566 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f639819a770 0x7f639819abf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6398109490 0x7f63981a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 --2- 172.21.15.63:0/3069719007 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f63981043c0 0x7f63981a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:03:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 >> 172.21.15.63:0/3069719007 conn(0x7f63980ffc20 msgr2=0x7f6398101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:03:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.563+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 shutdown_connections 2024-04-25T19:03:53.567 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:03:53.564+0000 7f639ec00640 1 -- 172.21.15.63:0/3069719007 wait complete. 2024-04-25T19:03:54.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 100221 23.33 MB/sec execute 112 sec latency 151.038 ms 2024-04-25T19:03:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:54 smithi114 ceph-mon[32988]: pgmap v549: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.4 MiB/s wr, 587 op/s 2024-04-25T19:03:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:54 smithi063 ceph-mon[28006]: pgmap v549: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.4 MiB/s wr, 587 op/s 2024-04-25T19:03:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:54 smithi070 ceph-mon[32820]: pgmap v549: 97 pgs: 97 active+clean; 848 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 8.4 MiB/s wr, 587 op/s 2024-04-25T19:03:55.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 100784 23.32 MB/sec execute 113 sec latency 54.633 ms 2024-04-25T19:03:56.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 101431 23.39 MB/sec execute 114 sec latency 52.430 ms 2024-04-25T19:03:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:56 smithi114 ceph-mon[32988]: pgmap v550: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-25T19:03:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:56 smithi063 ceph-mon[28006]: pgmap v550: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-25T19:03:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:56 smithi070 ceph-mon[32820]: pgmap v550: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 13 MiB/s wr, 954 op/s 2024-04-25T19:03:57.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 101929 23.47 MB/sec execute 115 sec latency 250.061 ms 2024-04-25T19:03:58.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 102703 23.60 MB/sec execute 116 sec latency 74.874 ms 2024-04-25T19:03:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:03:58 smithi114 ceph-mon[32988]: pgmap v551: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-25T19:03:58.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:03:58 smithi063 ceph-mon[28006]: pgmap v551: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-25T19:03:58.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:03:58 smithi070 ceph-mon[32820]: pgmap v551: 97 pgs: 97 active+clean; 854 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 10 MiB/s wr, 638 op/s 2024-04-25T19:03:59.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 103316 23.71 MB/sec execute 117 sec latency 50.854 ms 2024-04-25T19:03:59.506 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 14.0 2024-04-25T19:03:59.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:04:00.076 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:04:00.098 INFO:tasks.workunit.client.0.smithi063.stdout: 10 103907 23.77 MB/sec execute 118 sec latency 56.020 ms 2024-04-25T19:04:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:00 smithi114 ceph-mon[32988]: pgmap v552: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-25T19:04:00.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:00 smithi063 ceph-mon[28006]: pgmap v552: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-25T19:04:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:00 smithi070 ceph-mon[32820]: pgmap v552: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 921 op/s 2024-04-25T19:04:01.099 INFO:tasks.workunit.client.0.smithi063.stdout: 10 104423 23.77 MB/sec execute 119 sec latency 104.053 ms 2024-04-25T19:04:02.099 INFO:tasks.workunit.client.0.smithi063.stdout: 10 104760 23.71 MB/sec execute 120 sec latency 284.649 ms 2024-04-25T19:04:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:02 smithi114 ceph-mon[32988]: pgmap v553: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:02.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 -- 172.21.15.63:0/3509572614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc806a750 msgr2=0x7f1bc806abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 --2- 172.21.15.63:0/3509572614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc806a750 0x7f1bc806abb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009990 tx=0x7f1bb4009ca0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 -- 172.21.15.63:0/3509572614 shutdown_connections 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 --2- 172.21.15.63:0/3509572614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc806b0f0 0x7f1bc810f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 --2- 172.21.15.63:0/3509572614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc8108a30 0x7f1bc8108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 --2- 172.21.15.63:0/3509572614 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc806a750 0x7f1bc806abb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 -- 172.21.15.63:0/3509572614 >> 172.21.15.63:0/3509572614 conn(0x7f1bc80788f0 msgr2=0x7f1bc8078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 -- 172.21.15.63:0/3509572614 shutdown_connections 2024-04-25T19:04:02.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.828+0000 7f1bce200700 1 -- 172.21.15.63:0/3509572614 wait complete. 2024-04-25T19:04:02.830 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:02 smithi063 ceph-mon[28006]: pgmap v553: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:02.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.829+0000 7f1bce200700 1 Processor -- start 2024-04-25T19:04:02.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 -- start start 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc806b0f0 0x7f1bc8101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc8108a30 0x7f1bc8101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc8112c20 con 0x7f1bc806b0f0 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc8112aa0 con 0x7f1bc8108a30 2024-04-25T19:04:02.832 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.830+0000 7f1bce200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1bc8112da0 con 0x7f1bc8102490 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48784/0 (socket says 172.21.15.63:48784) 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 -- 172.21.15.63:0/3080039488 learned_addr learned my addr 172.21.15.63:0/3080039488 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7400700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc806b0f0 0x7f1bc8101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc6a00700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc8108a30 0x7f1bc8101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 -- 172.21.15.63:0/3080039488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc8108a30 msgr2=0x7f1bc8101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc8108a30 0x7f1bc8101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 -- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc806b0f0 msgr2=0x7f1bc8101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc806b0f0 0x7f1bc8101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.831+0000 7f1bc7e00700 1 -- 172.21.15.63:0/3080039488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bc81a7d20 con 0x7f1bc8102490 2024-04-25T19:04:02.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.832+0000 7f1bc7e00700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c370 tx=0x7f1bc000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:02.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.833+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc00092e0 con 0x7f1bc8102490 2024-04-25T19:04:02.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.833+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bc0003a00 con 0x7f1bc8102490 2024-04-25T19:04:02.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.833+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc00046f0 con 0x7f1bc8102490 2024-04-25T19:04:02.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.833+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bc81a7fb0 con 0x7f1bc8102490 2024-04-25T19:04:02.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.833+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bc81a83b0 con 0x7f1bc8102490 2024-04-25T19:04:02.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.835+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bc8069730 con 0x7f1bc8102490 2024-04-25T19:04:02.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.838+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1bc001a020 con 0x7f1bc8102490 2024-04-25T19:04:02.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.839+0000 7f1bbfe00700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 0x7f1bac0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:02.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.840+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8543+0+0 (secure 0 0 0) 0x7f1bc00cffe0 con 0x7f1bc8102490 2024-04-25T19:04:02.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.840+0000 7f1bc7400700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 0x7f1bac0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:02.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.841+0000 7f1bc7400700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 0x7f1bac0a63c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8005950 tx=0x7f1bb800b410 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:02.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:02.848+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bc0010040 con 0x7f1bc8102490 2024-04-25T19:04:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:02 smithi070 ceph-mon[32820]: pgmap v553: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:03.099 INFO:tasks.workunit.client.0.smithi063.stdout: 10 105244 23.74 MB/sec execute 121 sec latency 185.186 ms 2024-04-25T19:04:03.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.211+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bc802df80 con 0x7f1bc8102490 2024-04-25T19:04:03.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.212+0000 7f1bbfe00700 1 -- 172.21.15.63:0/3080039488 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+11871 (secure 0 0 0) 0x7f1bc009c610 con 0x7f1bc8102490 2024-04-25T19:04:03.214 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:04:03.214 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":54,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:03:40.454222+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24520,24533,24508]},"id":1}]} 2024-04-25T19:04:03.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 msgr2=0x7f1bac0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 0x7f1bac0a63c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8005950 tx=0x7f1bb800b410 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 msgr2=0x7f1bc81a75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c370 tx=0x7f1bc000c680 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 shutdown_connections 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1bac0a3f00 0x7f1bac0a63c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1bc8102490 0x7f1bc81a75b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1bc806b0f0 0x7f1bc8101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 --2- 172.21.15.63:0/3080039488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bc8108a30 0x7f1bc8101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 >> 172.21.15.63:0/3080039488 conn(0x7f1bc80788f0 msgr2=0x7f1bc80ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 shutdown_connections 2024-04-25T19:04:03.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:03.215+0000 7f1bce200700 1 -- 172.21.15.63:0/3080039488 wait complete. 2024-04-25T19:04:03.218 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 54 2024-04-25T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:03 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3080039488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:04:03.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:03 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3080039488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:04:03.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:03 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3080039488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:04:04.099 INFO:tasks.workunit.client.0.smithi063.stdout: 10 105957 23.92 MB/sec execute 122 sec latency 76.710 ms 2024-04-25T19:04:04.510 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db / --timeout 60 --expiration 74.0 --await-for 42.0' 2024-04-25T19:04:04.511 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db / --timeout 60 --expiration 74.0 --await-for 42.0 2024-04-25T19:04:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:04 smithi114 ceph-mon[32988]: pgmap v554: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:04 smithi063 ceph-mon[28006]: pgmap v554: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:04 smithi070 ceph-mon[32820]: pgmap v554: 97 pgs: 97 active+clean; 876 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.8 MiB/s wr, 651 op/s 2024-04-25T19:04:05.100 INFO:tasks.workunit.client.0.smithi063.stdout: 10 106421 23.95 MB/sec execute 123 sec latency 65.878 ms 2024-04-25T19:04:05.278 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:04:05.731 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:05 smithi063 ceph-mon[28006]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T19:04:05.731 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:05 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:05 smithi114 ceph-mon[32988]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T19:04:05.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:05 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:05.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:05 smithi070 ceph-mon[32820]: osdmap e124: 12 total, 12 up, 12 in 2024-04-25T19:04:05.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:05 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:06.100 INFO:tasks.workunit.client.0.smithi063.stdout: 10 106756 23.88 MB/sec execute 124 sec latency 129.053 ms 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 -- 172.21.15.63:0/3808282788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4354104b80 msgr2=0x7f4354108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 --2- 172.21.15.63:0/3808282788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4354104b80 0x7f4354108bc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4340009900 tx=0x7f434002a3f0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 -- 172.21.15.63:0/3808282788 shutdown_connections 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 --2- 172.21.15.63:0/3808282788 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f4354110e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 --2- 172.21.15.63:0/3808282788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4354104b80 0x7f4354108bc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 --2- 172.21.15.63:0/3808282788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 -- 172.21.15.63:0/3808282788 >> 172.21.15.63:0/3808282788 conn(0x7f43540ffa70 msgr2=0x7f4354101e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 -- 172.21.15.63:0/3808282788 shutdown_connections 2024-04-25T19:04:06.748 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.746+0000 7f435a000640 1 -- 172.21.15.63:0/3808282788 wait complete. 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.747+0000 7f435a000640 1 Processor -- start 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 -- start start 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f435419a650 0x7f435419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4354114210 con 0x7f435419a650 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4354114090 con 0x7f43541041d0 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f435a000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f4354114390 con 0x7f43541092a0 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f4353e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f4353e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48792/0 (socket says 172.21.15.63:48792) 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f4353e00640 1 -- 172.21.15.63:0/4228987047 learned_addr learned my addr 172.21.15.63:0/4228987047 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:06.751 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f4358c00640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.748+0000 7f4359600640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f435419a650 0x7f435419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 -- 172.21.15.63:0/4228987047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 msgr2=0x7f43541a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 -- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f435419a650 msgr2=0x7f435419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f435419a650 0x7f435419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 -- 172.21.15.63:0/4228987047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435419b250 con 0x7f43541092a0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4358c00640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4353e00640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4340005ab0 tx=0x7f4340003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4351600640 1 -- 172.21.15.63:0/4228987047 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434003f070 con 0x7f43541092a0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435419b4e0 con 0x7f43541092a0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4351600640 1 -- 172.21.15.63:0/4228987047 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4340031b50 con 0x7f43541092a0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.749+0000 7f4351600640 1 -- 172.21.15.63:0/4228987047 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4340035030 con 0x7f43541092a0 2024-04-25T19:04:06.752 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.750+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43541a7670 con 0x7f43541092a0 2024-04-25T19:04:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 Processor -- start 2024-04-25T19:04:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f4351600640 1 -- 172.21.15.63:0/4228987047 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4340042070 con 0x7f43541092a0 2024-04-25T19:04:06.754 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 -- start start 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd30fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd275900 con 0x55fbfd3102d0 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd21bef0 con 0x55fbfd30f9b0 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.753+0000 7f435e47e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd2718c0 con 0x55fbfd31ab90 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4358c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4359600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd30fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48796/0 (socket says 172.21.15.63:48796) 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 -- 172.21.15.63:0/2116273348 learned_addr learned my addr 172.21.15.63:0/2116273348 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 -- 172.21.15.63:0/2116273348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 msgr2=0x55fbfd30fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd30fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 -- 172.21.15.63:0/2116273348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 msgr2=0x55fbfd31a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 -- 172.21.15.63:0/2116273348 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfcf88210 con 0x55fbfd31ab90 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4353e00640 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43400ee3a0 tx=0x7f43400f4f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b877c7b69dd4e8a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.754+0000 7f4351600640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 0x7f433c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.755+0000 7f4351600640 1 -- 172.21.15.63:0/4228987047 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8631+0+0 (secure 0 0 0) 0x7f43400f0de0 con 0x7f43541092a0 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.755+0000 7f4359600640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 0x7f433c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.755+0000 7f4345c00640 1 -- 172.21.15.63:0/2116273348 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400bc6c0 con 0x55fbfd31ab90 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.755+0000 7f4345c00640 1 -- 172.21.15.63:0/2116273348 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43400bcd60 con 0x55fbfd31ab90 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.755+0000 7f4345c00640 1 -- 172.21.15.63:0/2116273348 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43400bc8a0 con 0x55fbfd31ab90 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f4359600640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 0x7f433c0a6250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f434c005fd0 tx=0x7f434c009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- 172.21.15.63:0/2116273348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 msgr2=0x55fbfd31cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f43400ee3a0 tx=0x7f43400f4f90 comp rx=0 tx=0).stop 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- 172.21.15.63:0/2116273348 shutdown_connections 2024-04-25T19:04:06.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd31cf80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 --2- 172.21.15.63:0/2116273348 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd30fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- 172.21.15.63:0/2116273348 >> 172.21.15.63:0/2116273348 conn(0x55fbfd30bec0 msgr2=0x55fbfd30c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- 172.21.15.63:0/2116273348 shutdown_connections 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- 172.21.15.63:0/2116273348 wait complete. 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 -- start start 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.756+0000 7f435e47e740 1 Processor -- start 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd3102d0 0x55fbfd317ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 0x55fbfd318530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd31d580 con 0x55fbfd30f9b0 2024-04-25T19:04:06.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd22f310 con 0x55fbfd31ab90 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55fbfd31d700 con 0x55fbfd3102d0 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f435e47e740 10 client.0 _ll_register_callbacks cb 0x55fbfd1fdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f4358c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f4353e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd3102d0 0x55fbfd317ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f4358c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38838/0 (socket says 172.21.15.63:38838) 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f4358c00640 1 -- 172.21.15.63:0/2984631127 learned_addr learned my addr 172.21.15.63:0/2984631127 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.757+0000 7f4359600640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 0x55fbfd318530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 -- 172.21.15.63:0/2984631127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd3102d0 msgr2=0x55fbfd317ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd3102d0 0x55fbfd317ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 -- 172.21.15.63:0/2984631127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 msgr2=0x55fbfd317ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 -- 172.21.15.63:0/2984631127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfd054620 con 0x55fbfd31ab90 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4358c00640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:06.763 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f4359600640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 0x55fbfd318530 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f434c003f50 tx=0x7f434c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f60c22acd5445b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f433a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c004b20 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43400bcd60 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.758+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8631+0+0 (secure 0 0 0) 0x7f434c004cc0 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f434c00f750 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 10 client.27200 fetch_fsmap learned FSMap version 55 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55fbfd329740 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43400bcd60 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7f434c026030 con 0x55fbfd31ab90 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 10 client.27200 fetch_fsmap finished waiting for FSMap version 55 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 10 client.27200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 20 client.27200 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd32aba0 0x55fbfd32cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 4 client.27200 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.759+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbfd2898b0 con 0x55fbfd32aba0 2024-04-25T19:04:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.760+0000 7f4358c00640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd32aba0 0x55fbfd32cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.760+0000 7f4358c00640 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd32aba0 0x55fbfd32cf90 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba8eabe4ef36abbf server_cookie=af5bfbedc2841c37 in_seq=0 out_seq=0 2024-04-25T19:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.760+0000 7f433a000640 10 client.27200 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f433a000640 1 -- 172.21.15.63:0/2984631127 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fbfd2898b0 con 0x55fbfd32aba0 2024-04-25T19:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f433a000640 10 client.27200 handle_command_reply: tid=0 2024-04-25T19:04:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 client.27200 shutdown 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 msgr2=0x55fbfd318530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 0x55fbfd318530 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f434c003f50 tx=0x7f434c004ab0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 shutdown_connections 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd32aba0 0x55fbfd32cf90 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd31ab90 0x55fbfd318530 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd3102d0 0x55fbfd317ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 --2- 172.21.15.63:0/2984631127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd317ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 >> 172.21.15.63:0/2984631127 conn(0x55fbfd30bec0 msgr2=0x55fbfd311c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f433a000640 0 client.27200 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.761+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 shutdown_connections 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.762+0000 7f435e47e740 1 -- 172.21.15.63:0/2984631127 wait complete. 2024-04-25T19:04:06.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.762+0000 7f435e47e740 20 client.27200 trim_cache size 0 max 16384 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.777+0000 7f435e47e740 1 Processor -- start 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.777+0000 7f435e47e740 1 -- start start 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd3263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd3102d0 0x55fbfd3123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd332860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f43541b26d0 con 0x55fbfd30f9b0 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43541aff10 con 0x55fbfd3102d0 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f435e47e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43541add80 con 0x55fbfd31ab90 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4358c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd3263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4358c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd3263b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38852/0 (socket says 172.21.15.63:38852) 2024-04-25T19:04:06.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd332860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4358c00640 1 -- 172.21.15.63:0/2706849422 learned_addr learned my addr 172.21.15.63:0/2706849422 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4353e00640 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd3102d0 0x55fbfd3123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 -- 172.21.15.63:0/2706849422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd3102d0 msgr2=0x55fbfd3123c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd3102d0 0x55fbfd3123c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 -- 172.21.15.63:0/2706849422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 msgr2=0x55fbfd3263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd3263b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.778+0000 7f4359600640 1 -- 172.21.15.63:0/2706849422 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfd312900 con 0x55fbfd31ab90 2024-04-25T19:04:06.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.779+0000 7f4359600640 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd332860 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f434c018040 tx=0x7f434c009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db15f1238ec04ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.779+0000 7f4345200640 1 -- 172.21.15.63:0/2706849422 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c033030 con 0x55fbfd31ab90 2024-04-25T19:04:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.779+0000 7f4345200640 1 -- 172.21.15.63:0/2706849422 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f434c005980 con 0x55fbfd31ab90 2024-04-25T19:04:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.779+0000 7f4345200640 1 -- 172.21.15.63:0/2706849422 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c0052b0 con 0x55fbfd31ab90 2024-04-25T19:04:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- 172.21.15.63:0/2706849422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 msgr2=0x55fbfd332860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:06.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd332860 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f434c018040 tx=0x7f434c009610 comp rx=0 tx=0).stop 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- 172.21.15.63:0/2706849422 shutdown_connections 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd3102d0 0x55fbfd3123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd332860 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 --2- 172.21.15.63:0/2706849422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd30f9b0 0x55fbfd3263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- 172.21.15.63:0/2706849422 >> 172.21.15.63:0/2706849422 conn(0x55fbfd30bec0 msgr2=0x55fbfd312e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- 172.21.15.63:0/2706849422 shutdown_connections 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- 172.21.15.63:0/2706849422 wait complete. 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 -- start start 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.780+0000 7f435e47e740 1 Processor -- start 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd38f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f43541ae3d0 con 0x55fbfd3102d0 2024-04-25T19:04:06.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f43541af840 con 0x55fbfd30f9b0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f43541aeaa0 con 0x55fbfd31ab90 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f435e47e740 10 client.0 _ll_register_callbacks cb 0x7f434c003de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4353e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4353e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38856/0 (socket says 172.21.15.63:38856) 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4353e00640 1 -- 172.21.15.63:0/3337327688 learned_addr learned my addr 172.21.15.63:0/3337327688 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4359600640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd38f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33774/0 (socket says 172.21.15.63:33774) 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 -- 172.21.15.63:0/3337327688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 msgr2=0x55fbfd38f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd38f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 -- 172.21.15.63:0/3337327688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 msgr2=0x55fbfd31fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4358c00640 1 -- 172.21.15.63:0/3337327688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfd38f610 con 0x55fbfd30f9b0 2024-04-25T19:04:06.786 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4353e00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.781+0000 7f4359600640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd38f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f4358c00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4348000c00 tx=0x7f434801fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cc10318d49d31f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f433a000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4348026070 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f434c005980 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8631+0+0 (secure 0 0 0) 0x7f4348005220 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f4348030050 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f435e47e740 10 client.27206 fetch_fsmap learned FSMap version 55 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55fbfd390780 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.782+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f434c005980 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7f4348022210 con 0x55fbfd30f9b0 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 10 client.27206 fetch_fsmap finished waiting for FSMap version 55 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 10 client.27206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:06.787 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 20 client.27206 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd390f10 0x55fbfd393330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 4 client.27206 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbfd283320 con 0x55fbfd390f10 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f4353e00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd390f10 0x55fbfd393330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f4353e00640 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd390f10 0x55fbfd393330 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4a97e4f94a48bb2 server_cookie=1853f5dc7f1541d8 in_seq=0 out_seq=0 2024-04-25T19:04:06.788 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:06.783+0000 7f433a000640 10 client.27206 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:06 smithi114 ceph-mon[32988]: pgmap v556: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:06 smithi063 ceph-mon[28006]: pgmap v556: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:06 smithi070 ceph-mon[32820]: pgmap v556: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:07.100 INFO:tasks.workunit.client.0.smithi063.stdout: 10 106932 23.78 MB/sec execute 125 sec latency 410.095 ms 2024-04-25T19:04:08.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 107374 23.80 MB/sec execute 126 sec latency 1058.902 ms 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 -- 172.21.15.63:0/1859075163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 msgr2=0x7f97e0103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 --2- 172.21.15.63:0/1859075163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e0103040 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009990 tx=0x7f97d4009ca0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 -- 172.21.15.63:0/1859075163 shutdown_connections 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 --2- 172.21.15.63:0/1859075163 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e0103040 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 --2- 172.21.15.63:0/1859075163 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e0104d70 0x7f97e010f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 --2- 172.21.15.63:0/1859075163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e01043b0 0x7f97e0104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 -- 172.21.15.63:0/1859075163 >> 172.21.15.63:0/1859075163 conn(0x7f97e00fe3a0 msgr2=0x7f97e01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:08.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 -- 172.21.15.63:0/1859075163 shutdown_connections 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.177+0000 7f97e6e00700 1 -- 172.21.15.63:0/1859075163 wait complete. 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.178+0000 7f97e6e00700 1 Processor -- start 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.178+0000 7f97e6e00700 1 -- start start 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e019b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e01043b0 0x7f97e019b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0104d70 0x7f97e019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0112bd0 con 0x7f97e0104d70 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0112a50 con 0x7f97e01043b0 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e6e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0112d50 con 0x7f97e0102c60 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e01043b0 0x7f97e019b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e01043b0 0x7f97e019b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41652/0 (socket says 172.21.15.63:41652) 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.179+0000 7f97e5000700 1 -- 172.21.15.63:0/2636907895 learned_addr learned my addr 172.21.15.63:0/2636907895 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:08.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e6400700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0104d70 0x7f97e019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5000700 1 -- 172.21.15.63:0/2636907895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 msgr2=0x7f97e019b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5a00700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e019b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5000700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e019b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5000700 1 -- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0104d70 msgr2=0x7f97e019fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5000700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0104d70 0x7f97e019fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5000700 1 -- 172.21.15.63:0/2636907895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e01a0150 con 0x7f97e01043b0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e6400700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0104d70 0x7f97e019fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.180+0000 7f97e5a00700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f97e0102c60 0x7f97e019b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97e5000700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97e01043b0 0x7f97e019b680 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00d980 tx=0x7f97cc00dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97de000700 1 -- 172.21.15.63:0/2636907895 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc016070 con 0x7f97e01043b0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97e6e00700 1 -- 172.21.15.63:0/2636907895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e01a0440 con 0x7f97e01043b0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97de000700 1 -- 172.21.15.63:0/2636907895 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97cc0044d0 con 0x7f97e01043b0 2024-04-25T19:04:08.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97de000700 1 -- 172.21.15.63:0/2636907895 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc004d00 con 0x7f97e01043b0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.181+0000 7f97e6e00700 1 -- 172.21.15.63:0/2636907895 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e01a7cc0 con 0x7f97e01043b0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.185+0000 7f97de000700 1 -- 172.21.15.63:0/2636907895 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f97cc004020 con 0x7f97e01043b0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.186+0000 7f97de000700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f97c80a3e50 0x7f97c80a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.186+0000 7f97f2cfcb80 1 Processor -- start 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.186+0000 7f97f2cfcb80 1 -- start start 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.186+0000 7f97de000700 1 -- 172.21.15.63:0/2636907895 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8631+0+0 (secure 0 0 0) 0x7f97cc0d0800 con 0x7f97e01043b0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.186+0000 7f97e5a00700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f97c80a3e50 0x7f97c80a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52ece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597d5235020 con 0x5597d52e1b80 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5597d51db8c0 con 0x5597d52e24a0 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5597d5230fe0 con 0x5597d52ed350 2024-04-25T19:04:08.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e5000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34944/0 (socket says 172.21.15.63:34944) 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51814/0 (socket says 172.21.15.63:51814) 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.187+0000 7f97e5a00700 1 -- 172.21.15.63:0/1090817934 learned_addr learned my addr 172.21.15.63:0/1090817934 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 -- 172.21.15.63:0/1090817934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 msgr2=0x5597d52ece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52ece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 -- 172.21.15.63:0/1090817934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 msgr2=0x5597d52e1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 -- 172.21.15.63:0/1090817934 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597d51d1810 con 0x5597d52ed350 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e5a00700 1 --2- 172.21.15.63:0/2636907895 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f97c80a3e50 0x7f97c80a6310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f97d4005190 tx=0x7f97d40051c0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97e6400700 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f97d80093a0 tx=0x7f97d8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=705deae60a5b53fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.188+0000 7f97d2a00700 1 -- 172.21.15.63:0/1090817934 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8019030 con 0x5597d52ed350 2024-04-25T19:04:08.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.189+0000 7f97d2a00700 1 -- 172.21.15.63:0/1090817934 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97d8004bb0 con 0x5597d52ed350 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.189+0000 7f97d2a00700 1 -- 172.21.15.63:0/1090817934 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8005020 con 0x5597d52ed350 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.189+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1090817934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 msgr2=0x5597d52ef740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.189+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f97d80093a0 tx=0x7f97d8009920 comp rx=0 tx=0).stop 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1090817934 shutdown_connections 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ef740 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e1b80 0x5597d52e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1090817934 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52ece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1090817934 >> 172.21.15.63:0/1090817934 conn(0x5597d5210730 msgr2=0x5597d52ddfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1090817934 shutdown_connections 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.190+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1090817934 wait complete. 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.191+0000 7f97f2cfcb80 1 -- start start 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.191+0000 7f97f2cfcb80 1 Processor -- start 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 0x5597d52e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597d52efed0 con 0x5597d52e24a0 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5597d51ee9f0 con 0x5597d52e1b80 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5597d52f0050 con 0x5597d52ed350 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97f2cfcb80 10 client.0 _ll_register_callbacks cb 0x5597d51cd8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e6400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:41666/0 (socket says 172.21.15.63:41666) 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 -- 172.21.15.63:0/1927830400 learned_addr learned my addr 172.21.15.63:0/1927830400 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5000700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 0x5597d52e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 -- 172.21.15.63:0/1927830400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 msgr2=0x5597d52ea230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ea230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 -- 172.21.15.63:0/1927830400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 msgr2=0x5597d52e9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 0x5597d52e9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:08.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5a00700 1 -- 172.21.15.63:0/1927830400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597d5182a60 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e6400700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.192+0000 7f97e5000700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 0x5597d52e9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009990 tx=0x7f97d4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=404dbb12dd33d367 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97c7400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4035060 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 8631+0+0 (secure 0 0 0) 0x7f97d400f460 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97d8004bb0 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f97d400f660 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97f2cfcb80 10 client.27218 fetch_fsmap learned FSMap version 55 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.193+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5597d51cbc80 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97d8004bb0 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7f97d4028890 con 0x5597d52e1b80 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 10 client.27218 fetch_fsmap finished waiting for FSMap version 55 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 10 client.27218 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:04:08.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 20 client.27218 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d52fcb10 0x5597d52fef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 4 client.27218 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597d5242810 con 0x5597d52fcb10 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.194+0000 7f97e5000700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d52fcb10 0x5597d52fef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.195+0000 7f97e5000700 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d52fcb10 0x5597d52fef60 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a73a6a8d3b6bac7b server_cookie=34901fd471f3aa24 in_seq=0 out_seq=0 2024-04-25T19:04:08.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:08.195+0000 7f97c7400700 10 client.27218 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:08 smithi114 ceph-mon[32988]: pgmap v557: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:08.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:08 smithi063 ceph-mon[28006]: pgmap v557: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:08.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:08 smithi070 ceph-mon[32820]: pgmap v557: 97 pgs: 97 active+clean; 899 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 890 op/s 2024-04-25T19:04:09.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 107736 23.74 MB/sec execute 127 sec latency 2058.956 ms 2024-04-25T19:04:10.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 108281 23.81 MB/sec execute 128 sec latency 3059.011 ms 2024-04-25T19:04:10.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:10 smithi063 ceph-mon[28006]: pgmap v558: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:10 smithi070 ceph-mon[32820]: pgmap v558: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:10 smithi114 ceph-mon[32988]: pgmap v558: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:11.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 108615 23.75 MB/sec execute 129 sec latency 4059.082 ms 2024-04-25T19:04:12.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 108873 23.67 MB/sec execute 130 sec latency 5059.178 ms 2024-04-25T19:04:12.868 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:12 smithi114 ceph-mon[32988]: pgmap v559: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:12.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:12 smithi063 ceph-mon[28006]: pgmap v559: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:12 smithi070 ceph-mon[32820]: pgmap v559: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:13.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109305 23.61 MB/sec execute 131 sec latency 6059.249 ms 2024-04-25T19:04:13.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:13 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:13.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:13 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:14.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:13 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:04:14.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109844 23.73 MB/sec execute 132 sec latency 7059.314 ms 2024-04-25T19:04:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.231+0000 7f433a000640 1 -- 172.21.15.63:0/3337327688 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fbfd283320 con 0x55fbfd390f10 2024-04-25T19:04:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.231+0000 7f433a000640 10 client.27206 handle_command_reply: tid=0 2024-04-25T19:04:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 client.27206 shutdown 2024-04-25T19:04:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 msgr2=0x55fbfd31f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4348000c00 tx=0x7f434801fa30 comp rx=0 tx=0).stop 2024-04-25T19:04:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 shutdown_connections 2024-04-25T19:04:14.234 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55fbfd390f10 0x55fbfd393330 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55fbfd31ab90 0x55fbfd38f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fbfd3102d0 0x55fbfd31fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 --2- 172.21.15.63:0/3337327688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fbfd30f9b0 0x55fbfd31f7f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f433a000640 0 client.27206 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 >> 172.21.15.63:0/3337327688 conn(0x55fbfd30bec0 msgr2=0x55fbfd317a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.233+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 shutdown_connections 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.233+0000 7f435e47e740 1 -- 172.21.15.63:0/3337327688 wait complete. 2024-04-25T19:04:14.235 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.233+0000 7f435e47e740 20 client.27206 trim_cache size 0 max 16384 2024-04-25T19:04:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:04:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:04:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:04:14.236 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:04:14.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.232+0000 7f97c7400700 1 -- 172.21.15.63:0/1927830400 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5597d5242810 con 0x5597d52fcb10 2024-04-25T19:04:14.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.233+0000 7f97c7400700 10 client.27218 handle_command_reply: tid=0 2024-04-25T19:04:14.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.233+0000 7f97f2cfcb80 1 client.27218 shutdown 2024-04-25T19:04:14.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 msgr2=0x5597d52ec9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:14.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f97d4009990 tx=0x7f97d4009ca0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 shutdown_connections 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d52fcb10 0x5597d52fef60 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52ed350 0x5597d52ea230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52ec9f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1927830400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52e24a0 0x5597d52e9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 >> 172.21.15.63:0/1927830400 conn(0x5597d5210730 msgr2=0x5597d52e3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:14.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.234+0000 7f97c7400700 0 client.27218 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.235+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 msgr2=0x7f433c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.235+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 0x7f433c0a6250 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f434c005fd0 tx=0x7f434c009500 comp rx=0 tx=0).stop 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.235+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 msgr2=0x7f43541a0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.235+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4340005ab0 tx=0x7f4340003b90 comp rx=0 tx=0).stop 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 shutdown_connections 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f433c0a3d90 0x7f433c0a6250 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f435419a650 0x7f435419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f43541092a0 0x7f43541a0ad0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.239 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 --2- 172.21.15.63:0/4228987047 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43541041d0 0x7f43541a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 >> 172.21.15.63:0/4228987047 conn(0x7f43540ffa70 msgr2=0x7f4354100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:14.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 shutdown_connections 2024-04-25T19:04:14.240 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.237+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 shutdown_connections 2024-04-25T19:04:14.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.236+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1927830400 wait complete. 2024-04-25T19:04:14.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.237+0000 7f97f2cfcb80 20 client.27218 trim_cache size 0 max 16384 2024-04-25T19:04:14.241 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:14.237+0000 7f435a000640 1 -- 172.21.15.63:0/4228987047 wait complete. 2024-04-25T19:04:14.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 Processor -- start 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 -- start start 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52e8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01b1da0 con 0x5597d52ed350 2024-04-25T19:04:14.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01af5a0 con 0x5597d52e1b80 2024-04-25T19:04:14.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.270+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ad410 con 0x5597d52e24a0 2024-04-25T19:04:14.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.271+0000 7f97e5000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.271+0000 7f97e5000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51826/0 (socket says 172.21.15.63:51826) 2024-04-25T19:04:14.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.271+0000 7f97e5000700 1 -- 172.21.15.63:0/3040168109 learned_addr learned my addr 172.21.15.63:0/3040168109 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:14.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.271+0000 7f97e5a00700 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52e8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.271+0000 7f97e6400700 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 -- 172.21.15.63:0/3040168109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 msgr2=0x5597d52e8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52e8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 -- 172.21.15.63:0/3040168109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 msgr2=0x5597d52f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 -- 172.21.15.63:0/3040168109 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597d52ff5e0 con 0x5597d52e24a0 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.272+0000 7f97e5000700 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00d950 tx=0x7f97cc0a0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e46677227498d3e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.273+0000 7f97d2000700 1 -- 172.21.15.63:0/3040168109 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0b2070 con 0x5597d52e24a0 2024-04-25T19:04:14.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.273+0000 7f97d2000700 1 -- 172.21.15.63:0/3040168109 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97cc0b5e50 con 0x5597d52e24a0 2024-04-25T19:04:14.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.273+0000 7f97d2000700 1 -- 172.21.15.63:0/3040168109 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc09c930 con 0x5597d52e24a0 2024-04-25T19:04:14.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/3040168109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 msgr2=0x5597d52f8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:14.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00d950 tx=0x7f97cc0a0040 comp rx=0 tx=0).stop 2024-04-25T19:04:14.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/3040168109 shutdown_connections 2024-04-25T19:04:14.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e1b80 0x5597d52e8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/3040168109 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e24a0 0x5597d52f8dc0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/3040168109 >> 172.21.15.63:0/3040168109 conn(0x5597d5210730 msgr2=0x5597d52eb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/3040168109 shutdown_connections 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.274+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/3040168109 wait complete. 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.275+0000 7f97f2cfcb80 1 -- start start 2024-04-25T19:04:14.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.275+0000 7f97f2cfcb80 1 Processor -- start 2024-04-25T19:04:14.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 0x5597d52f5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ada60 con 0x5597d52ed350 2024-04-25T19:04:14.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01aeed0 con 0x5597d52e24a0 2024-04-25T19:04:14.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01ae130 con 0x5597d52e1b80 2024-04-25T19:04:14.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97f2cfcb80 10 client.0 _ll_register_callbacks cb 0x5597d51cd8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 0x5597d52f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97e5a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 0x5597d52f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51832/0 (socket says 172.21.15.63:51832) 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97e5a00700 1 -- 172.21.15.63:0/1037593919 learned_addr learned my addr 172.21.15.63:0/1037593919 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97e5000700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.276+0000 7f97e6400700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 -- 172.21.15.63:0/1037593919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 msgr2=0x5597d52f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:14.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 0x5597d52f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 -- 172.21.15.63:0/1037593919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 msgr2=0x5597d52f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:14.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:14.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 -- 172.21.15.63:0/1037593919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597d52f4780 con 0x5597d52e24a0 2024-04-25T19:04:14.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97e5000700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52f3a40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00d950 tx=0x7f97cc0ce3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7f5749e07a3fdae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.277+0000 7f97c7400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0b9030 con 0x5597d52e24a0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f97cc00bb60 con 0x5597d52e24a0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97cc0b5e50 con 0x5597d52e24a0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f97cc0b2040 con 0x5597d52e24a0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97f2cfcb80 10 client.27224 fetch_fsmap learned FSMap version 55 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.278+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x5597d53640d0 con 0x5597d52e24a0 2024-04-25T19:04:14.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97cc0b5e50 con 0x5597d52e24a0 2024-04-25T19:04:14.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 55) ==== 8350+0+0 (secure 0 0 0) 0x7f97cc09c930 con 0x5597d52e24a0 2024-04-25T19:04:14.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 10 client.27224 fetch_fsmap finished waiting for FSMap version 55 2024-04-25T19:04:14.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 10 client.27224 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:04:14.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 20 client.27224 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:14.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d5364720 0x5597d5366b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:14.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 4 client.27224 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.0, "await_for": 42.0}] 2024-04-25T19:04:14.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.279+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 60.0, "expiration": 74.0, "await_for": 42.0}) -- 0x5597d523c8b0 con 0x5597d5364720 2024-04-25T19:04:14.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.280+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d5364720 0x5597d5366b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:14.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.280+0000 7f97e5a00700 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d5364720 0x5597d5366b40 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd49ed762c20645 server_cookie=3bb8c59cd08936f1 in_seq=0 out_seq=0 2024-04-25T19:04:14.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:14.280+0000 7f97c7400700 10 client.27224 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:14 smithi063 ceph-mon[28006]: pgmap v560: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:14.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:14 smithi063 ceph-mon[28006]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T19:04:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:14 smithi070 ceph-mon[32820]: pgmap v560: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:14 smithi070 ceph-mon[32820]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:14 smithi114 ceph-mon[32988]: pgmap v560: 97 pgs: 97 active+clean; 915 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 12 MiB/s wr, 954 op/s 2024-04-25T19:04:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:14 smithi114 ceph-mon[32988]: osdmap e125: 12 total, 12 up, 12 in 2024-04-25T19:04:15.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 23.56 MB/sec execute 133 sec latency 8059.432 ms 2024-04-25T19:04:15.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:15 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:15.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:15 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:15.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:15 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:04:15.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:15 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:15.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:15 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:15.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:15 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:04:16.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:15 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:04:16.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:15 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:04:16.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:15 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:04:16.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 23.38 MB/sec execute 134 sec latency 9059.536 ms 2024-04-25T19:04:16.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:16 smithi063 ceph-mon[28006]: pgmap v562: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:16 smithi070 ceph-mon[32820]: pgmap v562: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:17.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:16 smithi114 ceph-mon[32988]: pgmap v562: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:17.101 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 23.21 MB/sec execute 135 sec latency 10059.633 ms 2024-04-25T19:04:18.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 23.04 MB/sec execute 136 sec latency 11059.713 ms 2024-04-25T19:04:18.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:18 smithi063 ceph-mon[28006]: pgmap v563: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:18 smithi070 ceph-mon[32820]: pgmap v563: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:19.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:18 smithi114 ceph-mon[32988]: pgmap v563: 97 pgs: 97 active+clean; 938 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 663 op/s 2024-04-25T19:04:19.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.87 MB/sec execute 137 sec latency 12059.843 ms 2024-04-25T19:04:20.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.71 MB/sec execute 138 sec latency 13059.937 ms 2024-04-25T19:04:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:20 smithi063 ceph-mon[28006]: pgmap v564: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:20.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:20 smithi070 ceph-mon[32820]: pgmap v564: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:21.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:20 smithi114 ceph-mon[32988]: pgmap v564: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:21.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.54 MB/sec execute 139 sec latency 14060.064 ms 2024-04-25T19:04:22.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.38 MB/sec execute 140 sec latency 15060.150 ms 2024-04-25T19:04:22.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:22 smithi063 ceph-mon[28006]: pgmap v565: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:22 smithi070 ceph-mon[32820]: pgmap v565: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:23.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:22 smithi114 ceph-mon[32988]: pgmap v565: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:23.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.22 MB/sec execute 141 sec latency 16060.248 ms 2024-04-25T19:04:24.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 22.07 MB/sec execute 142 sec latency 17060.341 ms 2024-04-25T19:04:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:24 smithi063 ceph-mon[28006]: pgmap v566: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:24 smithi070 ceph-mon[32820]: pgmap v566: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:25.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:24 smithi114 ceph-mon[32988]: pgmap v566: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.5 MiB/s wr, 404 op/s 2024-04-25T19:04:25.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.91 MB/sec execute 143 sec latency 18060.505 ms 2024-04-25T19:04:25.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:25 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:25.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:25 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:26.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:25 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:26.102 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.76 MB/sec execute 144 sec latency 19060.589 ms 2024-04-25T19:04:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:26 smithi070 ceph-mon[32820]: pgmap v567: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.1 MiB/s wr, 320 op/s 2024-04-25T19:04:27.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:26 smithi114 ceph-mon[32988]: pgmap v567: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.1 MiB/s wr, 320 op/s 2024-04-25T19:04:27.103 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.61 MB/sec execute 145 sec latency 20060.725 ms 2024-04-25T19:04:27.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:26 smithi063 ceph-mon[28006]: pgmap v567: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 9.1 MiB/s wr, 320 op/s 2024-04-25T19:04:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.395+0000 7f8e7dc00640 1 -- 172.21.15.63:0/3307645349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e781092d0 msgr2=0x7f8e78110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.395+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/3307645349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e781092d0 0x7f8e78110ea0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009ca0 tx=0x7f8e7002a8c0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 -- 172.21.15.63:0/3307645349 shutdown_connections 2024-04-25T19:04:27.398 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/3307645349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e781092d0 0x7f8e78110ea0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/3307645349 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e78104bb0 0x7f8e78108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/3307645349 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 0x7f8e781045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 -- 172.21.15.63:0/3307645349 >> 172.21.15.63:0/3307645349 conn(0x7f8e780ffa60 msgr2=0x7f8e78101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 -- 172.21.15.63:0/3307645349 shutdown_connections 2024-04-25T19:04:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.396+0000 7f8e7dc00640 1 -- 172.21.15.63:0/3307645349 wait complete. 2024-04-25T19:04:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 Processor -- start 2024-04-25T19:04:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 -- start start 2024-04-25T19:04:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 0x7f8e7819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e7819f010 0x7f8e7819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114270 con 0x7f8e7819f010 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781140f0 con 0x7f8e78104200 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.397+0000 7f8e7dc00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781143f0 con 0x7f8e781092d0 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43126/0 (socket says 172.21.15.63:43126) 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e7819f010 0x7f8e7819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 -- 172.21.15.63:0/1284770641 learned_addr learned my addr 172.21.15.63:0/1284770641 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e77e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 0x7f8e7819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 -- 172.21.15.63:0/1284770641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 msgr2=0x7f8e7819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 0x7f8e7819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 -- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e7819f010 msgr2=0x7f8e7819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e7819f010 0x7f8e7819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 -- 172.21.15.63:0/1284770641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7819fb00 con 0x7f8e781092d0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.398+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000e9c0 tx=0x7f8e6000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.399+0000 7f8e75600640 1 -- 172.21.15.63:0/1284770641 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6000cd80 con 0x7f8e781092d0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.399+0000 7f8e75600640 1 -- 172.21.15.63:0/1284770641 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e600054d0 con 0x7f8e781092d0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.399+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7819fdf0 con 0x7f8e781092d0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.399+0000 7f8e75600640 1 -- 172.21.15.63:0/1284770641 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60010430 con 0x7f8e781092d0 2024-04-25T19:04:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.399+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e781a75e0 con 0x7f8e781092d0 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.402+0000 7f8e82054740 1 Processor -- start 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.402+0000 7f8e75600640 1 -- 172.21.15.63:0/1284770641 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8e60004fd0 con 0x7f8e781092d0 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.402+0000 7f8e82054740 1 -- start start 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.402+0000 7f8e82054740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.402+0000 7f8e82054740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 0x55f5118ed650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e82054740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 0x55f5118eff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e82054740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f511848900 con 0x55f5118e29b0 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e82054740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f5117eeef0 con 0x55f5118e32d0 2024-04-25T19:04:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e82054740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5118448c0 con 0x55f5118edb90 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60408/0 (socket says 172.21.15.63:60408) 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 -- 172.21.15.63:0/1213001169 learned_addr learned my addr 172.21.15.63:0/1213001169 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 0x55f5118eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 0x55f5118ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 -- 172.21.15.63:0/1213001169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 msgr2=0x55f5118eff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 0x55f5118eff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 -- 172.21.15.63:0/1213001169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 msgr2=0x55f5118ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 0x55f5118ed650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e7d200640 1 -- 172.21.15.63:0/1213001169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f51155b210 con 0x55f5118e29b0 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 0x55f5118eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.403+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 0x55f5118ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:27.407 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009ca0 tx=0x7f8e700093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efda3199fc99016 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e75600640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 0x7f8e5c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e6d600640 1 -- 172.21.15.63:0/1213001169 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70031cf0 con 0x55f5118e29b0 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e75600640 1 -- 172.21.15.63:0/1284770641 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f8e600d0c10 con 0x7f8e781092d0 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 0x7f8e5c0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e6d600640 1 -- 172.21.15.63:0/1213001169 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e70036030 con 0x55f5118e29b0 2024-04-25T19:04:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.404+0000 7f8e6d600640 1 -- 172.21.15.63:0/1213001169 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e700042a0 con 0x55f5118e29b0 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 -- 172.21.15.63:0/1213001169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 msgr2=0x55f5118e2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 0x7f8e5c0a61d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55f5118e3f50 tx=0x7f8e68005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009ca0 tx=0x7f8e700093e0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 -- 172.21.15.63:0/1213001169 shutdown_connections 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e32d0 0x55f5118ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118edb90 0x55f5118eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 --2- 172.21.15.63:0/1213001169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118e2d90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 -- 172.21.15.63:0/1213001169 >> 172.21.15.63:0/1213001169 conn(0x55f5118deec0 msgr2=0x55f5118df2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 -- 172.21.15.63:0/1213001169 shutdown_connections 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.405+0000 7f8e82054740 1 -- 172.21.15.63:0/1213001169 wait complete. 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.406+0000 7f8e82054740 1 -- start start 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.406+0000 7f8e82054740 1 Processor -- start 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 0x55f5118eab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.410 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118eb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f5118f05e0 con 0x55f5118edb90 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55f511802310 con 0x55f5118e29b0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55f5118f0760 con 0x55f5118e32d0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e82054740 10 client.0 _ll_register_callbacks cb 0x55f5117d0cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e77e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 0x55f5118eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60422/0 (socket says 172.21.15.63:60422) 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 -- 172.21.15.63:0/649243939 learned_addr learned my addr 172.21.15.63:0/649243939 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 -- 172.21.15.63:0/649243939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 msgr2=0x55f5118eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118eb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 -- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 msgr2=0x55f5118eab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 0x55f5118eab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e7d200640 1 -- 172.21.15.63:0/649243939 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f511627620 con 0x55f5118edb90 2024-04-25T19:04:27.411 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.407+0000 7f8e77e00640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 0x55f5118eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e7d200640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e7002ade0 tx=0x7f8e7002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910b449d31608cb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e66000640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e7004a040 con 0x55f5118edb90 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f8e70031650 con 0x55f5118edb90 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e70036030 con 0x55f5118edb90 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f8e7005e050 con 0x55f5118edb90 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e82054740 10 client.17007 fetch_fsmap learned FSMap version 57 2024-04-25T19:04:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.408+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55f5118f05e0 con 0x55f5118edb90 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e70036030 con 0x55f5118edb90 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 57) ==== 8342+0+0 (secure 0 0 0) 0x7f8e7004d030 con 0x55f5118edb90 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 10 client.17007 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 10 client.17007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 20 client.17007 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5118fda30 0x55f5118ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 4 client.17007 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.409+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f51185c8b0 con 0x55f5118fda30 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.410+0000 7f8e77e00640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5118fda30 0x55f5118ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.410+0000 7f8e77e00640 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5118fda30 0x55f5118ffe20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e030d503ccc936 server_cookie=9ce472dd40d2f901 in_seq=0 out_seq=0 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.410+0000 7f8e66000640 10 client.17007 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e66000640 1 -- 172.21.15.63:0/649243939 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f51185c8b0 con 0x55f5118fda30 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e66000640 10 client.17007 handle_command_reply: tid=0 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 client.17007 shutdown 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 msgr2=0x55f5118eb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8e7002ade0 tx=0x7f8e7002a8c0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 shutdown_connections 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f5118fda30 0x55f5118ffe20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118edb90 0x55f5118eb5a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 --2- 172.21.15.63:0/649243939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118e29b0 0x55f5118eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 >> 172.21.15.63:0/649243939 conn(0x55f5118deec0 msgr2=0x55f5118e4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.411+0000 7f8e66000640 0 client.17007 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.412+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 shutdown_connections 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.412+0000 7f8e82054740 1 -- 172.21.15.63:0/649243939 wait complete. 2024-04-25T19:04:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.412+0000 7f8e82054740 20 client.17007 trim_cache size 0 max 16384 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 Processor -- start 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 -- start start 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781b2520 con 0x55f5118e29b0 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.428+0000 7f8e82054740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781afdc0 con 0x55f5118edb90 2024-04-25T19:04:27.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e82054740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781adc30 con 0x55f5118e32d0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:43166/0 (socket says 172.21.15.63:43166) 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 -- 172.21.15.63:0/2432654424 learned_addr learned my addr 172.21.15.63:0/2432654424 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e7d200640 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e77e00640 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 -- 172.21.15.63:0/2432654424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 msgr2=0x55f5118eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 -- 172.21.15.63:0/2432654424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 msgr2=0x55f5118f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.429+0000 7f8e6fe00640 1 -- 172.21.15.63:0/2432654424 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f511900450 con 0x55f5118e32d0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8e6009caa0 tx=0x7f8e600d46c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a28b6924531c846 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e6cc00640 1 -- 172.21.15.63:0/2432654424 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6009c8e0 con 0x55f5118e32d0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e6cc00640 1 -- 172.21.15.63:0/2432654424 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e600046c0 con 0x55f5118e32d0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e6cc00640 1 -- 172.21.15.63:0/2432654424 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6009c470 con 0x55f5118e32d0 2024-04-25T19:04:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 -- 172.21.15.63:0/2432654424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 msgr2=0x55f5118e5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8e6009caa0 tx=0x7f8e600d46c0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 -- 172.21.15.63:0/2432654424 shutdown_connections 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118e5370 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 --2- 172.21.15.63:0/2432654424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.430+0000 7f8e82054740 1 -- 172.21.15.63:0/2432654424 >> 172.21.15.63:0/2432654424 conn(0x55f5118deec0 msgr2=0x55f5118eaae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.431+0000 7f8e82054740 1 -- 172.21.15.63:0/2432654424 shutdown_connections 2024-04-25T19:04:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.431+0000 7f8e82054740 1 -- 172.21.15.63:0/2432654424 wait complete. 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.431+0000 7f8e82054740 1 -- start start 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.431+0000 7f8e82054740 1 Processor -- start 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781ae280 con 0x55f5118e29b0 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781af6f0 con 0x55f5118edb90 2024-04-25T19:04:27.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8e781ae950 con 0x55f5118e32d0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e82054740 10 client.0 _ll_register_callbacks cb 0x7f8e70009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e77e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35184/0 (socket says 172.21.15.63:35184) 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 -- 172.21.15.63:0/1173504715 learned_addr learned my addr 172.21.15.63:0/1173504715 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e6fe00640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 -- 172.21.15.63:0/1173504715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 msgr2=0x55f5118f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 -- 172.21.15.63:0/1173504715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 msgr2=0x55f5118f5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e7d200640 1 -- 172.21.15.63:0/1173504715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5118f3a90 con 0x55f5118edb90 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.432+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e7d200640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8e70004af0 tx=0x7f8e70031450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e55af6645864224d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e66000640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:27.438 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70031640 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f511856320 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f8e700317e0 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f8e7002f610 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e82054740 10 client.27242 fetch_fsmap learned FSMap version 57 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.433+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55f511963dc0 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f511856320 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8342+0+0 (secure 0 0 0) 0x7f8e70068030 con 0x55f5118edb90 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 10 client.27242 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 10 client.27242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 20 client.27242 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f511964500 0x55f511966920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 4 client.27242 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f511850190 con 0x55f511964500 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.434+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f511964500 0x55f511966920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.435+0000 7f8e77e00640 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f511964500 0x55f511966920 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d38f3d7373e32d5 server_cookie=b03805569aef9073 in_seq=0 out_seq=0 2024-04-25T19:04:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.435+0000 7f8e66000640 10 client.27242 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e66000640 1 -- 172.21.15.63:0/1173504715 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f511850190 con 0x55f511964500 2024-04-25T19:04:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e66000640 10 client.27242 handle_command_reply: tid=0 2024-04-25T19:04:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 client.27242 shutdown 2024-04-25T19:04:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 msgr2=0x55f5118f34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8e70004af0 tx=0x7f8e70031450 comp rx=0 tx=0).stop 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 shutdown_connections 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55f511964500 0x55f511966920 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55f5118edb90 0x55f5118f34a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55f5118e32d0 0x55f5118f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 --2- 172.21.15.63:0/1173504715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f5118e29b0 0x55f5118f5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e66000640 0 client.27242 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 >> 172.21.15.63:0/1173504715 conn(0x55f5118deec0 msgr2=0x55f5118de020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.572+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 shutdown_connections 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e82054740 1 -- 172.21.15.63:0/1173504715 wait complete. 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e82054740 20 client.27242 trim_cache size 0 max 16384 2024-04-25T19:04:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 msgr2=0x7f8e5c0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 0x7f8e5c0a61d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x55f5118e3f50 tx=0x7f8e68005f90 comp rx=0 tx=0).stop 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 msgr2=0x7f8e7819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.573+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000e9c0 tx=0x7f8e6000ee80 comp rx=0 tx=0).stop 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 shutdown_connections 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f8e5c0a3d10 0x7f8e5c0a61d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e7819f010 0x7f8e7819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8e781092d0 0x7f8e7819ac90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 --2- 172.21.15.63:0/1284770641 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e78104200 0x7f8e7819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 >> 172.21.15.63:0/1284770641 conn(0x7f8e780ffa60 msgr2=0x7f8e78101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 shutdown_connections 2024-04-25T19:04:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:27.574+0000 7f8e7dc00640 1 -- 172.21.15.63:0/1284770641 wait complete. 2024-04-25T19:04:27.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:27 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:28.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:27 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:28.106 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.46 MB/sec execute 146 sec latency 21064.088 ms 2024-04-25T19:04:28.129 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:27 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:28.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:28 smithi070 ceph-mon[32820]: pgmap v568: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 6.7 MiB/s wr, 215 op/s 2024-04-25T19:04:29.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:28 smithi114 ceph-mon[32988]: pgmap v568: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 6.7 MiB/s wr, 215 op/s 2024-04-25T19:04:29.106 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.31 MB/sec execute 147 sec latency 22064.166 ms 2024-04-25T19:04:29.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:28 smithi063 ceph-mon[28006]: pgmap v568: 97 pgs: 97 active+clean; 957 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 6.7 MiB/s wr, 215 op/s 2024-04-25T19:04:30.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.17 MB/sec execute 148 sec latency 23064.938 ms 2024-04-25T19:04:30.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:30 smithi070 ceph-mon[32820]: pgmap v569: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.2 MiB/s wr, 358 op/s 2024-04-25T19:04:31.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:30 smithi114 ceph-mon[32988]: pgmap v569: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.2 MiB/s wr, 358 op/s 2024-04-25T19:04:31.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 21.03 MB/sec execute 149 sec latency 24065.102 ms 2024-04-25T19:04:31.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:30 smithi063 ceph-mon[28006]: pgmap v569: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 9.2 MiB/s wr, 358 op/s 2024-04-25T19:04:32.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.89 MB/sec execute 150 sec latency 25065.185 ms 2024-04-25T19:04:32.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:32 smithi063 ceph-mon[28006]: pgmap v570: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:32.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:32 smithi070 ceph-mon[32820]: pgmap v570: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:33.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:32 smithi114 ceph-mon[32988]: pgmap v570: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:33.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.75 MB/sec execute 151 sec latency 26065.257 ms 2024-04-25T19:04:34.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.61 MB/sec execute 152 sec latency 27065.395 ms 2024-04-25T19:04:34.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:34 smithi070 ceph-mon[32820]: pgmap v571: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:35.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:34 smithi114 ceph-mon[32988]: pgmap v571: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:35.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.48 MB/sec execute 153 sec latency 28065.534 ms 2024-04-25T19:04:35.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:34 smithi063 ceph-mon[28006]: pgmap v571: 97 pgs: 97 active+clean; 948 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 5.6 MiB/s wr, 237 op/s 2024-04-25T19:04:36.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.35 MB/sec execute 154 sec latency 29065.626 ms 2024-04-25T19:04:36.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:36 smithi070 ceph-mon[32820]: pgmap v572: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 6.6 MiB/s wr, 302 op/s 2024-04-25T19:04:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:36 smithi114 ceph-mon[32988]: pgmap v572: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 6.6 MiB/s wr, 302 op/s 2024-04-25T19:04:37.107 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.21 MB/sec execute 155 sec latency 30065.717 ms 2024-04-25T19:04:37.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:36 smithi063 ceph-mon[28006]: pgmap v572: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 6.6 MiB/s wr, 302 op/s 2024-04-25T19:04:38.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 20.08 MB/sec execute 156 sec latency 31065.879 ms 2024-04-25T19:04:38.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:38 smithi070 ceph-mon[32820]: pgmap v573: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.5 MiB/s wr, 209 op/s 2024-04-25T19:04:39.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:38 smithi114 ceph-mon[32988]: pgmap v573: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.5 MiB/s wr, 209 op/s 2024-04-25T19:04:39.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.96 MB/sec execute 157 sec latency 32065.976 ms 2024-04-25T19:04:39.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:38 smithi063 ceph-mon[28006]: pgmap v573: 97 pgs: 97 active+clean; 932 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 3.5 MiB/s wr, 209 op/s 2024-04-25T19:04:40.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.83 MB/sec execute 158 sec latency 33066.140 ms 2024-04-25T19:04:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.722+0000 7f25e0e00640 1 -- 172.21.15.63:0/3214864207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc101fc0 msgr2=0x7f25dc1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.722+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3214864207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc101fc0 0x7f25dc1023a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009be0 tx=0x7f25cc02a6c0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 -- 172.21.15.63:0/3214864207 shutdown_connections 2024-04-25T19:04:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3214864207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc107090 0x7f25dc10ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3214864207 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc102970 0x7f25dc1069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3214864207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc101fc0 0x7f25dc1023a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 -- 172.21.15.63:0/3214864207 >> 172.21.15.63:0/3214864207 conn(0x7f25dc0fd8a0 msgr2=0x7f25dc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 -- 172.21.15.63:0/3214864207 shutdown_connections 2024-04-25T19:04:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.723+0000 7f25e0e00640 1 -- 172.21.15.63:0/3214864207 wait complete. 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 Processor -- start 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 -- start start 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 0x7f25dc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 0x7f25dc1a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc111ff0 con 0x7f25dc074cd0 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc111e70 con 0x7f25dc107090 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.724+0000 7f25e0e00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc112170 con 0x7f25dc101fc0 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.728 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47448/0 (socket says 172.21.15.63:47448) 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 -- 172.21.15.63:0/3552534409 learned_addr learned my addr 172.21.15.63:0/3552534409 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25dbe00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 0x7f25dc1a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25daa00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 0x7f25dc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 -- 172.21.15.63:0/3552534409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 msgr2=0x7f25dc074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 0x7f25dc074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 -- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 msgr2=0x7f25dc1a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 0x7f25dc1a67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25db400640 1 -- 172.21.15.63:0/3552534409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc1a6ec0 con 0x7f25dc101fc0 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25dbe00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 0x7f25dc1a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.725+0000 7f25daa00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 0x7f25dc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25db400640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25cc02acb0 tx=0x7f25cc003b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25d3e00640 1 -- 172.21.15.63:0/3552534409 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc041020 con 0x7f25dc101fc0 2024-04-25T19:04:40.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc1a7150 con 0x7f25dc101fc0 2024-04-25T19:04:40.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25d3e00640 1 -- 172.21.15.63:0/3552534409 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25cc0318f0 con 0x7f25dc101fc0 2024-04-25T19:04:40.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25d3e00640 1 -- 172.21.15.63:0/3552534409 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc036040 con 0x7f25dc101fc0 2024-04-25T19:04:40.730 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.726+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc1a7690 con 0x7f25dc101fc0 2024-04-25T19:04:40.732 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.729+0000 7f25d3e00640 1 -- 172.21.15.63:0/3552534409 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f25cc044020 con 0x7f25dc101fc0 2024-04-25T19:04:40.732 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 Processor -- start 2024-04-25T19:04:40.732 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 -- start start 2024-04-25T19:04:40.733 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab86bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab878f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bfab7d1900 con 0x55bfab86c2d0 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bfab777ef0 con 0x55bfab876b90 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25e51a1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bfab7cd8c0 con 0x55bfab86b9b0 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25dbe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab86bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36718/0 (socket says 172.21.15.63:36718) 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.730+0000 7f25db400640 1 -- 172.21.15.63:0/1689491410 learned_addr learned my addr 172.21.15.63:0/1689491410 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25daa00640 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab878f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 -- 172.21.15.63:0/1689491410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 msgr2=0x55bfab878f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab878f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 -- 172.21.15.63:0/1689491410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 msgr2=0x55bfab876650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 -- 172.21.15.63:0/1689491410 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfab4e4210 con 0x55bfab86b9b0 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25d3e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 0x7f25c00a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25db400640 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25d3e00640 1 -- 172.21.15.63:0/3552534409 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f25cc0f12b0 con 0x7f25dc101fc0 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.731+0000 7f25dbe00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 0x7f25c00a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.732+0000 7f25dbe00640 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab86bd90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f25d4005fd0 tx=0x7f25d40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c883f415fc0f2e87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.732+0000 7f25d0c00640 1 -- 172.21.15.63:0/1689491410 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d40035f0 con 0x55bfab86b9b0 2024-04-25T19:04:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.732+0000 7f25d0c00640 1 -- 172.21.15.63:0/1689491410 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d400fe50 con 0x55bfab86b9b0 2024-04-25T19:04:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.732+0000 7f25d0c00640 1 -- 172.21.15.63:0/1689491410 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d401a960 con 0x55bfab86b9b0 2024-04-25T19:04:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- 172.21.15.63:0/1689491410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 msgr2=0x55bfab86bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25dbe00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 0x7f25c00a6280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f25d401f000 tx=0x7f25d4020000 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab86bd90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f25d4005fd0 tx=0x7f25d40118d0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- 172.21.15.63:0/1689491410 shutdown_connections 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab878f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab876650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1689491410 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab86bd90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- 172.21.15.63:0/1689491410 >> 172.21.15.63:0/1689491410 conn(0x55bfab867ec0 msgr2=0x55bfab8682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- 172.21.15.63:0/1689491410 shutdown_connections 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- 172.21.15.63:0/1689491410 wait complete. 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.733+0000 7f25e51a1740 1 -- start start 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 Processor -- start 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 0x55bfab873b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab876b90 0x55bfab874610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bfab8797c0 con 0x55bfab86c2d0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55bfab78b310 con 0x55bfab86b9b0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55bfab879940 con 0x55bfab876b90 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25e51a1740 10 client.0 _ll_register_callbacks cb 0x55bfab759cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.734+0000 7f25daa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25daa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36720/0 (socket says 172.21.15.63:36720) 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25daa00640 1 -- 172.21.15.63:0/4148884767 learned_addr learned my addr 172.21.15.63:0/4148884767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25dbe00640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab876b90 0x55bfab874610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 0x55bfab873b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 -- 172.21.15.63:0/4148884767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab876b90 msgr2=0x55bfab874610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab876b90 0x55bfab874610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 -- 172.21.15.63:0/4148884767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 msgr2=0x55bfab8740d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 -- 172.21.15.63:0/4148884767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfab5b0620 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25daa00640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25db400640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 0x55bfab873b90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f25cc000f80 tx=0x7f25cc0bb6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e5d76c15aca1bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25c9600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.735+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0bcf80 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f25cc0f1020 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d400fe50 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f25cc0bcb10 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 10 client.27254 fetch_fsmap learned FSMap version 57 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55bfab885a80 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d400fe50 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 57) ==== 8342+0+0 (secure 0 0 0) 0x7f25cc0eebe0 con 0x55bfab86b9b0 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 10 client.27254 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 10 client.27254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 20 client.27254 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab886f80 0x55bfab889440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 4 client.27254 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:04:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.736+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfab7e58b0 con 0x55bfab886f80 2024-04-25T19:04:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.737+0000 7f25daa00640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab886f80 0x55bfab889440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.737+0000 7f25daa00640 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab886f80 0x55bfab889440 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=749cd0e5f43bd458 server_cookie=4bedb9c698c99b5c in_seq=0 out_seq=0 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.737+0000 7f25c9600640 10 client.27254 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.738+0000 7f25c9600640 1 -- 172.21.15.63:0/4148884767 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfab7e58b0 con 0x55bfab886f80 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.738+0000 7f25c9600640 10 client.27254 handle_command_reply: tid=0 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 client.27254 shutdown 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 msgr2=0x55bfab873b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 0x55bfab873b90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f25cc000f80 tx=0x7f25cc0bb6c0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 shutdown_connections 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab886f80 0x55bfab889440 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab876b90 0x55bfab874610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab8740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.744 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 --2- 172.21.15.63:0/4148884767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab86b9b0 0x55bfab873b90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25c9600640 0 client.27254 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 >> 172.21.15.63:0/4148884767 conn(0x55bfab867ec0 msgr2=0x55bfab86dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 shutdown_connections 2024-04-25T19:04:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 1 -- 172.21.15.63:0/4148884767 wait complete. 2024-04-25T19:04:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.739+0000 7f25e51a1740 20 client.27254 trim_cache size 0 max 16384 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.755+0000 7f25e51a1740 1 Processor -- start 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 -- start start 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab882680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab88ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1b2600 con 0x55bfab86c2d0 2024-04-25T19:04:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1afea0 con 0x55bfab876b90 2024-04-25T19:04:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25e51a1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1add10 con 0x55bfab86b9b0 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25daa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25daa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36722/0 (socket says 172.21.15.63:36722) 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25daa00640 1 -- 172.21.15.63:0/1427030422 learned_addr learned my addr 172.21.15.63:0/1427030422 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25dbe00640 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab88ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.756+0000 7f25db400640 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab882680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 -- 172.21.15.63:0/1427030422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 msgr2=0x55bfab882680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab882680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 -- 172.21.15.63:0/1427030422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 msgr2=0x55bfab88ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab88ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 -- 172.21.15.63:0/1427030422 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfab889ac0 con 0x55bfab86c2d0 2024-04-25T19:04:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25daa00640 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55bfab889cf0 tx=0x7f25c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5164beff94dd4e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25cbe00640 1 -- 172.21.15.63:0/1427030422 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c40120d0 con 0x55bfab86c2d0 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25cbe00640 1 -- 172.21.15.63:0/1427030422 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c4012270 con 0x55bfab86c2d0 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.757+0000 7f25cbe00640 1 -- 172.21.15.63:0/1427030422 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c40047a0 con 0x55bfab86c2d0 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 -- 172.21.15.63:0/1427030422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 msgr2=0x55bfab86da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55bfab889cf0 tx=0x7f25c4005f90 comp rx=0 tx=0).stop 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 -- 172.21.15.63:0/1427030422 shutdown_connections 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab88ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86c2d0 0x55bfab86da70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 --2- 172.21.15.63:0/1427030422 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86b9b0 0x55bfab882680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 -- 172.21.15.63:0/1427030422 >> 172.21.15.63:0/1427030422 conn(0x55bfab867ec0 msgr2=0x55bfab86d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 -- 172.21.15.63:0/1427030422 shutdown_connections 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.758+0000 7f25e51a1740 1 -- 172.21.15.63:0/1427030422 wait complete. 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 -- start start 2024-04-25T19:04:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 Processor -- start 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab87cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1ae360 con 0x55bfab86b9b0 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1af7d0 con 0x55bfab876b90 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1aea30 con 0x55bfab86c2d0 2024-04-25T19:04:40.765 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25e51a1740 10 client.0 _ll_register_callbacks cb 0x7f25cc0ee900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.759+0000 7f25daa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36736/0 (socket says 172.21.15.63:36736) 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25db400640 1 -- 172.21.15.63:0/25545191 learned_addr learned my addr 172.21.15.63:0/25545191 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47476/0 (socket says 172.21.15.63:47476) 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25dbe00640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab87cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 -- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 msgr2=0x55bfab87cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab87cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 -- 172.21.15.63:0/25545191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 msgr2=0x55bfab87f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 -- 172.21.15.63:0/25545191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfab878f80 con 0x55bfab86c2d0 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25db400640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25daa00640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f25c4014010 tx=0x7f25c4012b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614565e7a1369770 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.760+0000 7f25c9600640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c4031020 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25c4012270 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f25c4004250 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f25c403c050 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25e51a1740 10 client.27271 fetch_fsmap learned FSMap version 57 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55bfab8ed540 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.761+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25c4012270 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 57) ==== 8342+0+0 (secure 0 0 0) 0x7f25c4003680 con 0x55bfab86c2d0 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 10 client.27271 fetch_fsmap finished waiting for FSMap version 57 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 10 client.27271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 20 client.27271 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab8edc00 0x55bfab8f0020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 4 client.27271 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:04:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfab7df320 con 0x55bfab8edc00 2024-04-25T19:04:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.762+0000 7f25db400640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab8edc00 0x55bfab8f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.763+0000 7f25db400640 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab8edc00 0x55bfab8f0020 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c5d31e7837a9439 server_cookie=385af3de551616ec in_seq=0 out_seq=0 2024-04-25T19:04:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.763+0000 7f25c9600640 10 client.27271 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.811+0000 7f25c9600640 1 -- 172.21.15.63:0/25545191 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfab7df320 con 0x55bfab8edc00 2024-04-25T19:04:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.811+0000 7f25c9600640 10 client.27271 handle_command_reply: tid=0 2024-04-25T19:04:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.811+0000 7f25e51a1740 1 client.27271 shutdown 2024-04-25T19:04:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.811+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 msgr2=0x55bfab87c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.814 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.811+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f25c4014010 tx=0x7f25c4012b80 comp rx=0 tx=0).stop 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 shutdown_connections 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55bfab8edc00 0x55bfab8f0020 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55bfab876b90 0x55bfab87cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55bfab86c2d0 0x55bfab87c420 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 --2- 172.21.15.63:0/25545191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bfab86b9b0 0x55bfab87f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25c9600640 0 client.27271 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 >> 172.21.15.63:0/25545191 conn(0x55bfab867ec0 msgr2=0x55bfab86dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 shutdown_connections 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 1 -- 172.21.15.63:0/25545191 wait complete. 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.812+0000 7f25e51a1740 20 client.27271 trim_cache size 0 max 16384 2024-04-25T19:04:40.815 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 msgr2=0x7f25c00a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 0x7f25c00a6280 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f25d401f000 tx=0x7f25d4020000 comp rx=0 tx=0).stop 2024-04-25T19:04:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 msgr2=0x7f25dc074250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25cc02acb0 tx=0x7f25cc003b80 comp rx=0 tx=0).stop 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 shutdown_connections 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f25c00a3dc0 0x7f25c00a6280 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f25dc107090 0x7f25dc074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25dc074cd0 0x7f25dc1a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 --2- 172.21.15.63:0/3552534409 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f25dc101fc0 0x7f25dc074250 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 >> 172.21.15.63:0/3552534409 conn(0x7f25dc0fd8a0 msgr2=0x7f25dc0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 shutdown_connections 2024-04-25T19:04:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:40.813+0000 7f25e0e00640 1 -- 172.21.15.63:0/3552534409 wait complete. 2024-04-25T19:04:40.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:40 smithi070 ceph-mon[32820]: pgmap v574: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 5.2 MiB/s wr, 262 op/s 2024-04-25T19:04:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:40 smithi114 ceph-mon[32988]: pgmap v574: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 5.2 MiB/s wr, 262 op/s 2024-04-25T19:04:41.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.71 MB/sec execute 159 sec latency 34066.272 ms 2024-04-25T19:04:41.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:40 smithi063 ceph-mon[28006]: pgmap v574: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 5.2 MiB/s wr, 262 op/s 2024-04-25T19:04:42.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.58 MB/sec execute 160 sec latency 35066.454 ms 2024-04-25T19:04:42.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:42 smithi070 ceph-mon[32820]: pgmap v575: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:43.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:42 smithi114 ceph-mon[32988]: pgmap v575: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:43.108 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.46 MB/sec execute 161 sec latency 36066.625 ms 2024-04-25T19:04:43.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:42 smithi063 ceph-mon[28006]: pgmap v575: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:44.109 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.34 MB/sec execute 162 sec latency 37066.775 ms 2024-04-25T19:04:44.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:44 smithi070 ceph-mon[32820]: pgmap v576: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:44 smithi114 ceph-mon[32988]: pgmap v576: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:45.109 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.22 MB/sec execute 163 sec latency 38066.948 ms 2024-04-25T19:04:45.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:44 smithi063 ceph-mon[28006]: pgmap v576: 97 pgs: 97 active+clean; 939 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 2.6 MiB/s wr, 119 op/s 2024-04-25T19:04:46.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 19.10 MB/sec execute 164 sec latency 39067.170 ms 2024-04-25T19:04:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:46 smithi070 ceph-mon[32820]: pgmap v577: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 4.1 MiB/s wr, 191 op/s 2024-04-25T19:04:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:46 smithi114 ceph-mon[32988]: pgmap v577: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 4.1 MiB/s wr, 191 op/s 2024-04-25T19:04:47.109 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.99 MB/sec execute 165 sec latency 40067.331 ms 2024-04-25T19:04:47.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:46 smithi063 ceph-mon[28006]: pgmap v577: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 4.1 MiB/s wr, 191 op/s 2024-04-25T19:04:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:47 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:47 smithi114 ceph-mon[32988]: pgmap v578: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 3.2 MiB/s wr, 125 op/s 2024-04-25T19:04:48.109 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.87 MB/sec execute 166 sec latency 41067.427 ms 2024-04-25T19:04:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:47 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:47 smithi063 ceph-mon[28006]: pgmap v578: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 3.2 MiB/s wr, 125 op/s 2024-04-25T19:04:48.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:47 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:04:48.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:47 smithi070 ceph-mon[32820]: pgmap v578: 97 pgs: 97 active+clean; 928 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 3.2 MiB/s wr, 125 op/s 2024-04-25T19:04:49.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.76 MB/sec execute 167 sec latency 42067.565 ms 2024-04-25T19:04:50.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.65 MB/sec execute 168 sec latency 43067.730 ms 2024-04-25T19:04:50.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:50 smithi070 ceph-mon[32820]: pgmap v579: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.6 MiB/s wr, 152 op/s 2024-04-25T19:04:50.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:50 smithi114 ceph-mon[32988]: pgmap v579: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.6 MiB/s wr, 152 op/s 2024-04-25T19:04:50.831 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:50 smithi063 ceph-mon[28006]: pgmap v579: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 3.6 MiB/s wr, 152 op/s 2024-04-25T19:04:51.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.54 MB/sec execute 169 sec latency 44067.864 ms 2024-04-25T19:04:52.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.43 MB/sec execute 170 sec latency 45068.066 ms 2024-04-25T19:04:52.703 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:52 smithi070 ceph-mon[32820]: pgmap v580: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:52.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:52 smithi114 ceph-mon[32988]: pgmap v580: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:52.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:52 smithi063 ceph-mon[28006]: pgmap v580: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:53.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.32 MB/sec execute 171 sec latency 46068.233 ms 2024-04-25T19:04:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 -- 172.21.15.63:0/1761563412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 msgr2=0x7fc90c105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 --2- 172.21.15.63:0/1761563412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c105b00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009890 tx=0x7fc8f802a360 comp rx=0 tx=0).stop 2024-04-25T19:04:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 -- 172.21.15.63:0/1761563412 shutdown_connections 2024-04-25T19:04:53.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 --2- 172.21.15.63:0/1761563412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c106230 0x7fc90c1066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 --2- 172.21.15.63:0/1761563412 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c105b00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.961+0000 7fc913000640 1 --2- 172.21.15.63:0/1761563412 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c100d20 0x7fc90c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.962+0000 7fc913000640 1 -- 172.21.15.63:0/1761563412 >> 172.21.15.63:0/1761563412 conn(0x7fc90c0fc770 msgr2=0x7fc90c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.962+0000 7fc913000640 1 -- 172.21.15.63:0/1761563412 shutdown_connections 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.962+0000 7fc913000640 1 -- 172.21.15.63:0/1761563412 wait complete. 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.963+0000 7fc913000640 1 Processor -- start 2024-04-25T19:04:53.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.963+0000 7fc913000640 1 -- start start 2024-04-25T19:04:53.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 0x7fc90c1a48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c1a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 0x7fc90c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1185d0 con 0x7fc90c100d20 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c118450 con 0x7fc90c106230 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc913000640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c118750 con 0x7fc90c101640 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 0x7fc90c1a48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc912600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 0x7fc90c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 0x7fc90c1a48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45060/0 (socket says 172.21.15.63:45060) 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911c00640 1 -- 172.21.15.63:0/2598288769 learned_addr learned my addr 172.21.15.63:0/2598288769 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911200640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c1a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc912600640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 0x7fc90c19ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45038/0 (socket says 172.21.15.63:45038) 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911200640 1 -- 172.21.15.63:0/2598288769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 msgr2=0x7fc90c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:53.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911200640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 0x7fc90c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911200640 1 -- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 msgr2=0x7fc90c1a48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.964+0000 7fc911200640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 0x7fc90c1a48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc911200640 1 -- 172.21.15.63:0/2598288769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c19f1d0 con 0x7fc90c101640 2024-04-25T19:04:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc911200640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c1a4de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc8f802a950 tx=0x7fc8f8032d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc90a000640 1 -- 172.21.15.63:0/2598288769 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8040020 con 0x7fc90c101640 2024-04-25T19:04:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90c19f460 con 0x7fc90c101640 2024-04-25T19:04:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc90c1ab760 con 0x7fc90c101640 2024-04-25T19:04:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc90a000640 1 -- 172.21.15.63:0/2598288769 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8f8030d20 con 0x7fc90c101640 2024-04-25T19:04:53.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.965+0000 7fc90a000640 1 -- 172.21.15.63:0/2598288769 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8037040 con 0x7fc90c101640 2024-04-25T19:04:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 Processor -- start 2024-04-25T19:04:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 -- start start 2024-04-25T19:04:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc90a000640 1 -- 172.21.15.63:0/2598288769 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc8f8043020 con 0x7fc90c101640 2024-04-25T19:04:53.972 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc51a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc51ab90 0x563ecc51cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563ecc475900 con 0x563ecc50f9b0 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563ecc41bef0 con 0x563ecc5102d0 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.969+0000 7fc91741d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563ecc4718c0 con 0x563ecc51ab90 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45068/0 (socket says 172.21.15.63:45068) 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 -- 172.21.15.63:0/4194952029 learned_addr learned my addr 172.21.15.63:0/4194952029 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc911c00640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc51a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc911200640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc51ab90 0x563ecc51cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 -- 172.21.15.63:0/4194952029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc51ab90 msgr2=0x563ecc51cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc51ab90 0x563ecc51cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 -- 172.21.15.63:0/4194952029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 msgr2=0x563ecc51a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:53.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc51a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc912600640 1 -- 172.21.15.63:0/4194952029 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ecc188210 con 0x563ecc50f9b0 2024-04-25T19:04:53.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.970+0000 7fc911c00640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc51a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc912600640 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc904009be0 tx=0x7fc904011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5726bffc0572d2d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc8fea00640 1 -- 172.21.15.63:0/4194952029 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904004230 con 0x563ecc50f9b0 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc8fea00640 1 -- 172.21.15.63:0/4194952029 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9040043d0 con 0x563ecc50f9b0 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc8fea00640 1 -- 172.21.15.63:0/4194952029 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90400e990 con 0x563ecc50f9b0 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc90a000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 0x7fc8f40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc90a000640 1 -- 172.21.15.63:0/2598288769 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fc8f80efd80 con 0x7fc90c101640 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.971+0000 7fc911c00640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 0x7fc8f40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 -- 172.21.15.63:0/4194952029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 msgr2=0x563ecc50fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc904009be0 tx=0x7fc904011440 comp rx=0 tx=0).stop 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 -- 172.21.15.63:0/4194952029 shutdown_connections 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc51a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc51ab90 0x563ecc51cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 --2- 172.21.15.63:0/4194952029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc50f9b0 0x563ecc50fd90 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 -- 172.21.15.63:0/4194952029 >> 172.21.15.63:0/4194952029 conn(0x563ecc50bec0 msgr2=0x563ecc50c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 -- 172.21.15.63:0/4194952029 shutdown_connections 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc91741d740 1 -- 172.21.15.63:0/4194952029 wait complete. 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.972+0000 7fc911c00640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 0x7fc8f40a6250 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563ecc510f50 tx=0x7fc900009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 -- start start 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 Processor -- start 2024-04-25T19:04:53.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc517aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc517fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc518520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563ecc51d7c0 con 0x563ecc51ab90 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563ecc42f310 con 0x563ecc50f9b0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563ecc51d940 con 0x563ecc5102d0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.973+0000 7fc91741d740 10 client.0 _ll_register_callbacks cb 0x563ecc3fdcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc911200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc517fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc518520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc911200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc517fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:47604/0 (socket says 172.21.15.63:47604) 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc911200640 1 -- 172.21.15.63:0/3098458302 learned_addr learned my addr 172.21.15.63:0/3098458302 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc911c00640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc517aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 -- 172.21.15.63:0/3098458302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 msgr2=0x563ecc517fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc517fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 -- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 msgr2=0x563ecc517aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc517aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 -- 172.21.15.63:0/3098458302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ecc254620 con 0x563ecc51ab90 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc912600640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc518520 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc90401b040 tx=0x7fc9040038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d29d8f1f81d3d94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.974+0000 7fc8f3400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904027070 con 0x563ecc51ab90 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fc904005250 con 0x563ecc51ab90 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9040043d0 con 0x563ecc51ab90 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc904033050 con 0x563ecc51ab90 2024-04-25T19:04:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc91741d740 10 client.17043 fetch_fsmap learned FSMap version 58 2024-04-25T19:04:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563ecc5299a0 con 0x563ecc51ab90 2024-04-25T19:04:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.975+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9040043d0 con 0x563ecc51ab90 2024-04-25T19:04:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7fc904004b70 con 0x563ecc51ab90 2024-04-25T19:04:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 10 client.17043 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 20 client.17043 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc52adb0 0x563ecc52d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 4 client.17043 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ecc4898b0 con 0x563ecc52adb0 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.976+0000 7fc911c00640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc52adb0 0x563ecc52d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.977+0000 7fc911c00640 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc52adb0 0x563ecc52d1a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b2cd8bdcde03e73 server_cookie=304c777c973ff454 in_seq=0 out_seq=0 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.977+0000 7fc8f3400640 10 client.17043 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.977+0000 7fc8f3400640 1 -- 172.21.15.63:0/3098458302 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563ecc4898b0 con 0x563ecc52adb0 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.977+0000 7fc8f3400640 10 client.17043 handle_command_reply: tid=0 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 client.17043 shutdown 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 msgr2=0x563ecc518520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc518520 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc90401b040 tx=0x7fc9040038a0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 shutdown_connections 2024-04-25T19:04:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc52adb0 0x563ecc52d1a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc518520 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc517fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 --2- 172.21.15.63:0/3098458302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc517aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc8f3400640 0 client.17043 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 >> 172.21.15.63:0/3098458302 conn(0x563ecc50bec0 msgr2=0x563ecc511c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 shutdown_connections 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 1 -- 172.21.15.63:0/3098458302 wait complete. 2024-04-25T19:04:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.978+0000 7fc91741d740 20 client.17043 trim_cache size 0 max 16384 2024-04-25T19:04:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.995+0000 7fc91741d740 1 Processor -- start 2024-04-25T19:04:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 -- start start 2024-04-25T19:04:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc50f9b0 0x563ecc526680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.998 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc532910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b6960 con 0x563ecc51ab90 2024-04-25T19:04:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b4230 con 0x563ecc5102d0 2024-04-25T19:04:53.999 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc91741d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b20a0 con 0x563ecc50f9b0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc911200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc532910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc911c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc50f9b0 0x563ecc526680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc911200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45076/0 (socket says 172.21.15.63:45076) 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.996+0000 7fc911200640 1 -- 172.21.15.63:0/2317939996 learned_addr learned my addr 172.21.15.63:0/2317939996 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 -- 172.21.15.63:0/2317939996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc50f9b0 msgr2=0x563ecc526680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc50f9b0 0x563ecc526680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 -- 172.21.15.63:0/2317939996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 msgr2=0x563ecc532910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc532910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 -- 172.21.15.63:0/2317939996 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ecc507820 con 0x563ecc5102d0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc911200640 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80f35d0 tx=0x7fc8f80dc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac53c860afc40b59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:54.000 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc8fe000640 1 -- 172.21.15.63:0/2317939996 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80f1040 con 0x563ecc5102d0 2024-04-25T19:04:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc8fe000640 1 -- 172.21.15.63:0/2317939996 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8f80bc490 con 0x563ecc5102d0 2024-04-25T19:04:54.001 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.997+0000 7fc8fe000640 1 -- 172.21.15.63:0/2317939996 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80bbd90 con 0x563ecc5102d0 2024-04-25T19:04:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 -- 172.21.15.63:0/2317939996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 msgr2=0x563ecc50aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc8f80f35d0 tx=0x7fc8f80dc040 comp rx=0 tx=0).stop 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 -- 172.21.15.63:0/2317939996 shutdown_connections 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc5102d0 0x563ecc50aea0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc532910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 --2- 172.21.15.63:0/2317939996 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc50f9b0 0x563ecc526680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 -- 172.21.15.63:0/2317939996 >> 172.21.15.63:0/2317939996 conn(0x563ecc50bec0 msgr2=0x563ecc50a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 -- 172.21.15.63:0/2317939996 shutdown_connections 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.998+0000 7fc91741d740 1 -- 172.21.15.63:0/2317939996 wait complete. 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 -- start start 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 Processor -- start 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc520880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc520dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b26f0 con 0x563ecc51ab90 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b3b60 con 0x563ecc50f9b0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c1b2dc0 con 0x563ecc5102d0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:53.999+0000 7fc91741d740 10 client.0 _ll_register_callbacks cb 0x7fc904005a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc911200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc520dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45082/0 (socket says 172.21.15.63:45082) 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 -- 172.21.15.63:0/2661844775 learned_addr learned my addr 172.21.15.63:0/2661844775 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc911c00640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc520880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 -- 172.21.15.63:0/2661844775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 msgr2=0x563ecc520dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc520dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 -- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 msgr2=0x563ecc520880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc520880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 -- 172.21.15.63:0/2661844775 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ecc5904b0 con 0x563ecc51ab90 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc911200640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc520dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:54.005 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc911c00640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc520880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc912600640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc904000c00 tx=0x7fc904003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01f5075ef3c65f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.000+0000 7fc8f3400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90403b030 con 0x563ecc51ab90 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fc90400edf0 con 0x563ecc51ab90 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8f80bc490 con 0x563ecc51ab90 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fc904047050 con 0x563ecc51ab90 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc91741d740 10 client.17049 fetch_fsmap learned FSMap version 58 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563ecc591760 con 0x563ecc51ab90 2024-04-25T19:04:54.006 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.001+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8f80bc490 con 0x563ecc51ab90 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7fc904005490 con 0x563ecc51ab90 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 10 client.17049 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 10 client.17049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 20 client.17049 populate_metadata read hostname 'smithi063' 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc591df0 0x563ecc594210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 4 client.17049 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ecc483320 con 0x563ecc591df0 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.002+0000 7fc911c00640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc591df0 0x563ecc594210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.003+0000 7fc911c00640 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc591df0 0x563ecc594210 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a49cc42acdb529d3 server_cookie=304ec389c4d4d8be in_seq=0 out_seq=0 2024-04-25T19:04:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.003+0000 7fc8f3400640 10 client.17049 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc8f3400640 1 -- 172.21.15.63:0/2661844775 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ecc483320 con 0x563ecc591df0 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc8f3400640 10 client.17049 handle_command_reply: tid=0 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 client.17049 shutdown 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 msgr2=0x563ecc521320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc904000c00 tx=0x7fc904003eb0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 shutdown_connections 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563ecc591df0 0x563ecc594210 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ecc51ab90 0x563ecc521320 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563ecc5102d0 0x563ecc520dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 --2- 172.21.15.63:0/2661844775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563ecc50f9b0 0x563ecc520880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc8f3400640 0 client.17049 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 >> 172.21.15.63:0/2661844775 conn(0x563ecc50bec0 msgr2=0x563ecc511bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 shutdown_connections 2024-04-25T19:04:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.037+0000 7fc91741d740 1 -- 172.21.15.63:0/2661844775 wait complete. 2024-04-25T19:04:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc91741d740 20 client.17049 trim_cache size 0 max 16384 2024-04-25T19:04:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:04:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:04:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:04:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 msgr2=0x7fc8f40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 0x7fc8f40a6250 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x563ecc510f50 tx=0x7fc900009500 comp rx=0 tx=0).stop 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 msgr2=0x7fc90c1a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c1a4de0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc8f802a950 tx=0x7fc8f8032d30 comp rx=0 tx=0).stop 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 shutdown_connections 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc8f40a3d90 0x7fc8f40a6250 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc90c106230 0x7fc90c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc90c101640 0x7fc90c1a4de0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 --2- 172.21.15.63:0/2598288769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc90c100d20 0x7fc90c1a48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.038+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 >> 172.21.15.63:0/2598288769 conn(0x7fc90c0fc770 msgr2=0x7fc90c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.039+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 shutdown_connections 2024-04-25T19:04:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:04:54.039+0000 7fc913000640 1 -- 172.21.15.63:0/2598288769 wait complete. 2024-04-25T19:04:54.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.22 MB/sec execute 172 sec latency 47068.320 ms 2024-04-25T19:04:54.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:54 smithi070 ceph-mon[32820]: pgmap v581: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:54 smithi114 ceph-mon[32988]: pgmap v581: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:54.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:54 smithi063 ceph-mon[28006]: pgmap v581: 97 pgs: 97 active+clean; 922 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-04-25T19:04:55.110 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.11 MB/sec execute 173 sec latency 48068.423 ms 2024-04-25T19:04:56.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 18.01 MB/sec execute 174 sec latency 49068.564 ms 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.282+0000 7f97c7400700 1 -- 172.21.15.63:0/1037593919 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: -115 ) ==== 8+0+594 (crc 0 0 0) 0x5597d523c8b0 con 0x5597d5364720 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.282+0000 7f97c7400700 10 client.27224 handle_command_reply: tid=0 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.282+0000 7f97f2cfcb80 1 client.27224 shutdown 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 msgr2=0x5597d52f3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52f3a40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f97cc00d950 tx=0x7f97cc0ce3e0 comp rx=0 tx=0).stop 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 shutdown_connections 2024-04-25T19:04:56.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5597d5364720 0x5597d5366b40 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5597d52e1b80 0x5597d52f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597d52ed350 0x5597d52f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 --2- 172.21.15.63:0/1037593919 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5597d52e24a0 0x5597d52f3a40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97c7400700 0 client.27224 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.283+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 >> 172.21.15.63:0/1037593919 conn(0x5597d5210730 msgr2=0x5597d52eb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.284+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 shutdown_connections 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.284+0000 7f97f2cfcb80 1 -- 172.21.15.63:0/1037593919 wait complete. 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:04:56.284+0000 7f97f2cfcb80 20 client.27224 trim_cache size 0 max 16384 2024-04-25T19:04:56.289 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 58, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 49, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "e43e1e63": { 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 49, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 0.0, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCING", 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 42.0 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 74.0, 2024-04-25T19:04:56.290 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCING", 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 42.0 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:04:56.291 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINPROGRESS: 2024-04-25T19:04:56.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:56 smithi063 ceph-mon[28006]: pgmap v582: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.0 MiB/s wr, 115 op/s 2024-04-25T19:04:56.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:56 smithi114 ceph-mon[32988]: pgmap v582: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.0 MiB/s wr, 115 op/s 2024-04-25T19:04:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:56 smithi070 ceph-mon[32820]: pgmap v582: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 2.0 MiB/s wr, 115 op/s 2024-04-25T19:04:57.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.90 MB/sec execute 175 sec latency 50068.670 ms 2024-04-25T19:04:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 115 2024-04-25T19:04:57.273 WARNING:tasks.quiescer.fs.[cephfs]:Set 'e43e1e63' hasn't quiesced after 57.8 seconds (timeout: 60). Dumping ops with locks from all ranks. 2024-04-25T19:04:57.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:04:57.772 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:04:58.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.80 MB/sec execute 176 sec latency 51068.764 ms 2024-04-25T19:04:58.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:04:58 smithi063 ceph-mon[28006]: pgmap v583: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 485 KiB/s wr, 43 op/s 2024-04-25T19:04:58.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:04:58 smithi114 ceph-mon[32988]: pgmap v583: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 485 KiB/s wr, 43 op/s 2024-04-25T19:04:58.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:04:58 smithi070 ceph-mon[32820]: pgmap v583: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 485 KiB/s wr, 43 op/s 2024-04-25T19:04:59.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.70 MB/sec execute 177 sec latency 52068.885 ms 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 -- 172.21.15.63:0/1487212940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 msgr2=0x7f54c0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 --2- 172.21.15.63:0/1487212940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0102f10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f54ac009990 tx=0x7f54ac009ca0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 -- 172.21.15.63:0/1487212940 shutdown_connections 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 --2- 172.21.15.63:0/1487212940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 0x7f54c010f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 --2- 172.21.15.63:0/1487212940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c0108e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 --2- 172.21.15.63:0/1487212940 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0102f10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 -- 172.21.15.63:0/1487212940 >> 172.21.15.63:0/1487212940 conn(0x7f54c0079ba0 msgr2=0x7f54c0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 -- 172.21.15.63:0/1487212940 shutdown_connections 2024-04-25T19:05:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.102+0000 7f54c7600700 1 -- 172.21.15.63:0/1487212940 wait complete. 2024-04-25T19:05:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.104+0000 7f54c7600700 1 Processor -- start 2024-04-25T19:05:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.104+0000 7f54c7600700 1 -- start start 2024-04-25T19:05:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.104+0000 7f54c7600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c7600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 0x7f54c0075280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c7600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c00757c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c7600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0112c50 con 0x7f54c0108a50 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c7600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0112ad0 con 0x7f54c0103450 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c7600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f54c0112dd0 con 0x7f54c0102a90 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c5600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c00757c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51296/0 (socket says 172.21.15.63:51296) 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 -- 172.21.15.63:0/3032430241 learned_addr learned my addr 172.21.15.63:0/3032430241 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54b7e00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 0x7f54c0075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 -- 172.21.15.63:0/3032430241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 msgr2=0x7f54c0075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 0x7f54c0075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 -- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 msgr2=0x7f54c00757c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c00757c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c4c00700 1 -- 172.21.15.63:0/3032430241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c01a7680 con 0x7f54c0102a90 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.105+0000 7f54c5600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54c4c00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f54b000e9c0 tx=0x7f54b000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b000cbd0 con 0x7f54c0102a90 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54b000ee60 con 0x7f54c0102a90 2024-04-25T19:05:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b00055c0 con 0x7f54c0102a90 2024-04-25T19:05:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c01a7970 con 0x7f54c0102a90 2024-04-25T19:05:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.106+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c01a7e60 con 0x7f54c0102a90 2024-04-25T19:05:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.107+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54c0069730 con 0x7f54c0102a90 2024-04-25T19:05:00.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.60 MB/sec execute 178 sec latency 53068.973 ms 2024-04-25T19:05:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.110+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f54b0004750 con 0x7f54c0102a90 2024-04-25T19:05:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.111+0000 7f54be000700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 0x7f54a40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.111+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f54b00d0c10 con 0x7f54c0102a90 2024-04-25T19:05:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.111+0000 7f54b7e00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 0x7f54a40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.112+0000 7f54b7e00700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 0x7f54a40a62f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54c007a280 tx=0x7f54ac005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:00.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.113+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f54b0053ff0 con 0x7f54c0102a90 2024-04-25T19:05:00.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.398+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54c002df80 con 0x7f54c0102a90 2024-04-25T19:05:00.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.399+0000 7f54be000700 1 -- 172.21.15.63:0/3032430241 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11865 (secure 0 0 0) 0x7f54b009d210 con 0x7f54c0102a90 2024-04-25T19:05:00.402 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:05:00.402 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:04:46.191966+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 msgr2=0x7f54a40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 0x7f54a40a62f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f54c007a280 tx=0x7f54ac005fd0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 msgr2=0x7f54c0074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f54b000e9c0 tx=0x7f54b000ecd0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 shutdown_connections 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f54a40a3e30 0x7f54a40a62f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f54c0102a90 0x7f54c0074d40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54c0108a50 0x7f54c00757c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 --2- 172.21.15.63:0/3032430241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54c0103450 0x7f54c0075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:00.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.402+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 >> 172.21.15.63:0/3032430241 conn(0x7f54c0079ba0 msgr2=0x7f54c01018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:00.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.404+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 shutdown_connections 2024-04-25T19:05:00.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:00.404+0000 7f54c7600700 1 -- 172.21.15.63:0/3032430241 wait complete. 2024-04-25T19:05:00.407 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 58 2024-04-25T19:05:00.797 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:00 smithi063 ceph-mon[28006]: pgmap v584: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 492 KiB/s wr, 47 op/s 2024-04-25T19:05:00.797 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:00 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3032430241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:00 smithi114 ceph-mon[32988]: pgmap v584: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 492 KiB/s wr, 47 op/s 2024-04-25T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:00 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3032430241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:00 smithi070 ceph-mon[32820]: pgmap v584: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 492 KiB/s wr, 47 op/s 2024-04-25T19:05:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:00 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3032430241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:01.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.50 MB/sec execute 179 sec latency 54069.114 ms 2024-04-25T19:05:01.273 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 2 (c) to a remote file /var/run/ceph/1/ops-e43e1e63-1-mds.c.json 2024-04-25T19:05:01.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.1:2 ops --flags=locks --path=/var/run/ceph/ops-e43e1e63-1-mds.c.json 2024-04-25T19:05:01.763 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:05:02.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.41 MB/sec execute 180 sec latency 55069.206 ms 2024-04-25T19:05:02.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:02 smithi114 ceph-mon[32988]: pgmap v585: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:02.863 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:02 smithi063 ceph-mon[28006]: pgmap v585: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:02.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:02 smithi070 ceph-mon[32820]: pgmap v585: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:03.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.31 MB/sec execute 181 sec latency 56069.373 ms 2024-04-25T19:05:04.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.22 MB/sec execute 182 sec latency 57069.460 ms 2024-04-25T19:05:04.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.125+0000 7f8513e00700 1 -- 172.21.15.63:0/1031488946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f851410f2d0 msgr2=0x7f85141116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.125+0000 7f8513e00700 1 --2- 172.21.15.63:0/1031488946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f851410f2d0 0x7f85141116c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f850c00b0a0 tx=0x7f850c00b3b0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 -- 172.21.15.63:0/1031488946 shutdown_connections 2024-04-25T19:05:04.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 --2- 172.21.15.63:0/1031488946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f851410f2d0 0x7f85141116c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 --2- 172.21.15.63:0/1031488946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 0x7f8514101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 --2- 172.21.15.63:0/1031488946 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f85141020d0 0x7f851410ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 -- 172.21.15.63:0/1031488946 >> 172.21.15.63:0/1031488946 conn(0x7f85140fd360 msgr2=0x7f85140ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 -- 172.21.15.63:0/1031488946 shutdown_connections 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.126+0000 7f8513e00700 1 -- 172.21.15.63:0/1031488946 wait complete. 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 Processor -- start 2024-04-25T19:05:04.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 -- start start 2024-04-25T19:05:04.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 0x7f851419b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 0x7f851419faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8514114d40 con 0x7f85141020d0 2024-04-25T19:05:04.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8514114bc0 con 0x7f8514101720 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.127+0000 7f8513e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8514114ec0 con 0x7f851410f2d0 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39062/0 (socket says 172.21.15.63:39062) 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 -- 172.21.15.63:0/4262769594 learned_addr learned my addr 172.21.15.63:0/4262769594 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512a00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 0x7f851419b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8513400700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 0x7f851419faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 -- 172.21.15.63:0/4262769594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 msgr2=0x7f851419faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 0x7f851419faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 -- 172.21.15.63:0/4262769594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 msgr2=0x7f851419b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 0x7f851419b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8512000700 1 -- 172.21.15.63:0/4262769594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85141a01e0 con 0x7f85141020d0 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.128+0000 7f8513400700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 0x7f851419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f8512000700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f850000eb30 tx=0x7f850000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f850b400700 1 -- 172.21.15.63:0/4262769594 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850000cbd0 con 0x7f85141020d0 2024-04-25T19:05:04.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f850b400700 1 -- 172.21.15.63:0/4262769594 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85000044d0 con 0x7f85141020d0 2024-04-25T19:05:04.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f850b400700 1 -- 172.21.15.63:0/4262769594 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500003e30 con 0x7f85141020d0 2024-04-25T19:05:04.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85141a04d0 con 0x7f85141020d0 2024-04-25T19:05:04.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.129+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85141a7d50 con 0x7f85141020d0 2024-04-25T19:05:04.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.132+0000 7f850b400700 1 -- 172.21.15.63:0/4262769594 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8500004000 con 0x7f85141020d0 2024-04-25T19:05:04.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.134+0000 7f850b400700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 0x7f84f80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.134+0000 7f8524374b80 1 Processor -- start 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.134+0000 7f850b400700 1 -- 172.21.15.63:0/4262769594 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f85000d0b00 con 0x7f85141020d0 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.134+0000 7f8524374b80 1 -- start start 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.134+0000 7f8512a00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 0x7f84f80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe0c5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55affe00af70 con 0x55affe0b83f0 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55affdfb1810 con 0x55affe0b7ad0 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8524374b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55affe006f30 con 0x55affe0c32a0 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8512000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8512a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8512000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39078/0 (socket says 172.21.15.63:39078) 2024-04-25T19:05:04.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8512000700 1 -- 172.21.15.63:0/632898814 learned_addr learned my addr 172.21.15.63:0/632898814 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.135+0000 7f8513400700 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe0c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 0x7f84f80a62f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8504009cb0 tx=0x7f8504009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 -- 172.21.15.63:0/632898814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 msgr2=0x55affe0c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe0c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 -- 172.21.15.63:0/632898814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 msgr2=0x55affe0c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 -- 172.21.15.63:0/632898814 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affdfa7760 con 0x55affe0b7ad0 2024-04-25T19:05:04.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512000700 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:04.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f8512a00700 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0b7eb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8504011cc0 tx=0x7f850401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c029906cd0ee9a76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f84ffe00700 1 -- 172.21.15.63:0/632898814 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504004230 con 0x55affe0b7ad0 2024-04-25T19:05:04.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f84ffe00700 1 -- 172.21.15.63:0/632898814 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850401cdd0 con 0x55affe0b7ad0 2024-04-25T19:05:04.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.136+0000 7f84ffe00700 1 -- 172.21.15.63:0/632898814 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504005020 con 0x55affe0b7ad0 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 -- 172.21.15.63:0/632898814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 msgr2=0x55affe0b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0b7eb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8504011cc0 tx=0x7f850401c000 comp rx=0 tx=0).stop 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 -- 172.21.15.63:0/632898814 shutdown_connections 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe0c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0b7eb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 --2- 172.21.15.63:0/632898814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 -- 172.21.15.63:0/632898814 >> 172.21.15.63:0/632898814 conn(0x55affdfe6680 msgr2=0x55affe0b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 -- 172.21.15.63:0/632898814 shutdown_connections 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.137+0000 7f8524374b80 1 -- 172.21.15.63:0/632898814 wait complete. 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.138+0000 7f8524374b80 1 -- start start 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.138+0000 7f8524374b80 1 Processor -- start 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b83f0 0x55affe0bfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 0x55affe0c0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55affe0c5e20 con 0x55affe0c32a0 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55affdfc4940 con 0x55affe0b7ad0 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55affe0c5fa0 con 0x55affe0b83f0 2024-04-25T19:05:04.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8524374b80 10 client.0 _ll_register_callbacks cb 0x55affdfa3800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:04.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8512a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8512a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45610/0 (socket says 172.21.15.63:45610) 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8512a00700 1 -- 172.21.15.63:0/3775520565 learned_addr learned my addr 172.21.15.63:0/3775520565 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8513400700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 0x55affe0c0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.139+0000 7f8512000700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b83f0 0x55affe0bfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 -- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b83f0 msgr2=0x55affe0bfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b83f0 0x55affe0bfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 -- 172.21.15.63:0/3775520565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 msgr2=0x55affe0c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 -- 172.21.15.63:0/3775520565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affdf589b0 con 0x55affe0c32a0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8513400700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 0x55affe0c0200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f850c00d010 tx=0x7f850c003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f586bf18eccce3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f84fcc00700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c0078f0 con 0x55affe0c32a0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f850c007a90 con 0x55affe0c32a0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.140+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f850401cdd0 con 0x55affe0c32a0 2024-04-25T19:05:04.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f850c007c90 con 0x55affe0c32a0 2024-04-25T19:05:04.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f8524374b80 10 client.17067 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:04.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55affdfa1bd0 con 0x55affe0c32a0 2024-04-25T19:05:04.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f850401cdd0 con 0x55affe0c32a0 2024-04-25T19:05:04.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f850c02b030 con 0x55affe0c32a0 2024-04-25T19:05:04.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f8524374b80 10 client.17067 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:04.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.141+0000 7f8524374b80 10 client.17067 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.142+0000 7f8524374b80 20 client.17067 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.142+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe0d2ab0 0x55affe0d4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.142+0000 7f8524374b80 4 client.17067 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.142+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55affe018760 con 0x55affe0d2ab0 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.142+0000 7f8512a00700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe0d2ab0 0x55affe0d4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.143+0000 7f8512a00700 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe0d2ab0 0x55affe0d4ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f195181d20739952 server_cookie=6fb8cc6bdc08db1 in_seq=0 out_seq=0 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.143+0000 7f84fcc00700 10 client.17067 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.143+0000 7f84fcc00700 1 -- 172.21.15.63:0/3775520565 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55affe018760 con 0x55affe0d2ab0 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.143+0000 7f84fcc00700 10 client.17067 handle_command_reply: tid=0 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.143+0000 7f8524374b80 1 client.17067 shutdown 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 msgr2=0x55affe0c0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 0x55affe0c0200 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f850c00d010 tx=0x7f850c003c30 comp rx=0 tx=0).stop 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 shutdown_connections 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe0d2ab0 0x55affe0d4ea0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0c32a0 0x55affe0c0200 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b7ad0 0x55affe0c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 --2- 172.21.15.63:0/3775520565 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b83f0 0x55affe0bfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 >> 172.21.15.63:0/3775520565 conn(0x55affdfe6680 msgr2=0x55affe0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f84fcc00700 0 client.17067 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 shutdown_connections 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 1 -- 172.21.15.63:0/3775520565 wait complete. 2024-04-25T19:05:04.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.144+0000 7f8524374b80 20 client.17067 trim_cache size 0 max 16384 2024-04-25T19:05:04.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.164+0000 7f8524374b80 1 Processor -- start 2024-04-25T19:05:04.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 -- start start 2024-04-25T19:05:04.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0da750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0c32a0 0x55affe0dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f85141b1c70 con 0x55affe0b83f0 2024-04-25T19:05:04.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85141af500 con 0x55affe0c32a0 2024-04-25T19:05:04.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8524374b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ad370 con 0x55affe0b7ad0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8512a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8512a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51340/0 (socket says 172.21.15.63:51340) 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8512a00700 1 -- 172.21.15.63:0/1940326691 learned_addr learned my addr 172.21.15.63:0/1940326691 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8513400700 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0c32a0 0x55affe0dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.165+0000 7f8512000700 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0da750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 -- 172.21.15.63:0/1940326691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0c32a0 msgr2=0x55affe0dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0c32a0 0x55affe0dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 -- 172.21.15.63:0/1940326691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 msgr2=0x55affe0da750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0da750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 -- 172.21.15.63:0/1940326691 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55affe0b9630 con 0x55affe0b7ad0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f8512a00700 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8504011fd0 tx=0x7f8504009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebcc922bd7fd0c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f84ff400700 1 -- 172.21.15.63:0/1940326691 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850401cd80 con 0x55affe0b7ad0 2024-04-25T19:05:04.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f84ff400700 1 -- 172.21.15.63:0/1940326691 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8504036180 con 0x55affe0b7ad0 2024-04-25T19:05:04.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.166+0000 7f84ff400700 1 -- 172.21.15.63:0/1940326691 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85040038a0 con 0x55affe0b7ad0 2024-04-25T19:05:04.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 -- 172.21.15.63:0/1940326691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 msgr2=0x55affe0ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8504011fd0 tx=0x7f8504009710 comp rx=0 tx=0).stop 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 -- 172.21.15.63:0/1940326691 shutdown_connections 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0b7ad0 0x55affe0ad2a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0c32a0 0x55affe0dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 --2- 172.21.15.63:0/1940326691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b83f0 0x55affe0da750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 -- 172.21.15.63:0/1940326691 >> 172.21.15.63:0/1940326691 conn(0x55affdfe6680 msgr2=0x55affe0acb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 -- 172.21.15.63:0/1940326691 shutdown_connections 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.167+0000 7f8524374b80 1 -- 172.21.15.63:0/1940326691 wait complete. 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 -- start start 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 Processor -- start 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 0x55affe0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ad9c0 con 0x55affe0b7ad0 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f85141aee30 con 0x55affe0b83f0 2024-04-25T19:05:04.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.168+0000 7f8524374b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f85141ae090 con 0x55affe0c32a0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8524374b80 10 client.0 _ll_register_callbacks cb 0x55affdfa3800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45636/0 (socket says 172.21.15.63:45636) 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 -- 172.21.15.63:0/536573941 learned_addr learned my addr 172.21.15.63:0/536573941 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512a00700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 0x55affe0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8513400700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 -- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 msgr2=0x55affe137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 -- 172.21.15.63:0/536573941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 msgr2=0x55affe0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 0x55affe0c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 -- 172.21.15.63:0/536573941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55affe0ae140 con 0x55affe0b83f0 2024-04-25T19:05:04.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512a00700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 0x55affe0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f8512000700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f850009c700 tx=0x7f850009c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21f7562762cce68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.169+0000 7f84fcc00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85000b5030 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f850009ce60 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8504036180 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f8500080050 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 10 client.27302 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55affe139660 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8504036180 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f85000a51e0 con 0x55affe0b83f0 2024-04-25T19:05:04.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 10 client.27302 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 10 client.27302 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.170+0000 7f8524374b80 20 client.27302 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe139d80 0x55affe13c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f8524374b80 4 client.27302 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.c.json"}] 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 --> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.c.json"}) -- 0x55affe012800 con 0x55affe139d80 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f8512a00700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe139d80 0x55affe13c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f8512a00700 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe139d80 0x55affe13c1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d4e4cfd4a08bbf6 server_cookie=f557b74dce2bb458 in_seq=0 out_seq=0 2024-04-25T19:05:04.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.171+0000 7f84fcc00700 10 client.27302 ms_handle_connect on v2:172.21.15.114:6836/1526734616 2024-04-25T19:05:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.193+0000 7f84fcc00700 1 -- 172.21.15.63:0/536573941 <== mds.2 v2:172.21.15.114:6836/1526734616 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55affe012800 con 0x55affe139d80 2024-04-25T19:05:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.193+0000 7f84fcc00700 10 client.27302 handle_command_reply: tid=0 2024-04-25T19:05:04.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.193+0000 7f8524374b80 1 client.27302 shutdown 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.193+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 msgr2=0x55affe0c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.193+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f850009c700 tx=0x7f850009c360 comp rx=0 tx=0).stop 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 shutdown_connections 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:6836/1526734616,v1:172.21.15.114:6837/1526734616] conn(0x55affe139d80 0x55affe13c1a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55affe0c32a0 0x55affe137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55affe0b7ad0 0x55affe0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 --2- 172.21.15.63:0/536573941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55affe0b83f0 0x55affe0c98b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 >> 172.21.15.63:0/536573941 conn(0x55affdfe6680 msgr2=0x55affe0c14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f84fcc00700 0 client.27302 ms_handle_reset on v2:172.21.15.114:6836/1526734616 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 shutdown_connections 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 1 -- 172.21.15.63:0/536573941 wait complete. 2024-04-25T19:05:04.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.194+0000 7f8524374b80 20 client.27302 trim_cache size 0 max 16384 2024-04-25T19:05:04.197 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:05:04.197 INFO:teuthology.orchestra.run.smithi063.stdout: "path": "/var/run/ceph/ops-e43e1e63-1-mds.c.json" 2024-04-25T19:05:04.197 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:05:04.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 msgr2=0x7f84f80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 0x7f84f80a62f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8504009cb0 tx=0x7f8504009260 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 msgr2=0x7f851419b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f850000eb30 tx=0x7f850000ee40 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 shutdown_connections 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f84f80a3e30 0x7f84f80a62f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f851410f2d0 0x7f851419faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8514101720 0x7f851419b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 --2- 172.21.15.63:0/4262769594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f85141020d0 0x7f851419b710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.195+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 >> 172.21.15.63:0/4262769594 conn(0x7f85140fd360 msgr2=0x7f8514110590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.196+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 shutdown_connections 2024-04-25T19:05:04.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:04.196+0000 7f8513e00700 1 -- 172.21.15.63:0/4262769594 wait complete. 2024-04-25T19:05:04.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:04 smithi114 ceph-mon[32988]: pgmap v586: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:04.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:04 smithi063 ceph-mon[28006]: pgmap v586: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:04 smithi070 ceph-mon[32820]: pgmap v586: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 79 KiB/s wr, 20 op/s 2024-04-25T19:05:05.097 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 1 (i) to a remote file /var/run/ceph/1/ops-e43e1e63-1-mds.i.json 2024-04-25T19:05:05.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.1:1 ops --flags=locks --path=/var/run/ceph/ops-e43e1e63-1-mds.i.json 2024-04-25T19:05:05.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.12 MB/sec execute 183 sec latency 58069.549 ms 2024-04-25T19:05:05.560 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:05:06.111 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 17.03 MB/sec execute 184 sec latency 59069.663 ms 2024-04-25T19:05:06.756 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:06 smithi063 ceph-mon[28006]: pgmap v587: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 107 KiB/s wr, 39 op/s 2024-04-25T19:05:06.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:06 smithi114 ceph-mon[32988]: pgmap v587: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 107 KiB/s wr, 39 op/s 2024-04-25T19:05:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:06 smithi070 ceph-mon[32820]: pgmap v587: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 107 KiB/s wr, 39 op/s 2024-04-25T19:05:07.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.94 MB/sec execute 185 sec latency 60069.786 ms 2024-04-25T19:05:07.188 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 -- 172.21.15.63:0/3399093745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a41043b0 msgr2=0x7f74a4104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 --2- 172.21.15.63:0/3399093745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a41043b0 0x7f74a4104790 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f749402a410 comp rx=0 tx=0).stop 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 -- 172.21.15.63:0/3399093745 shutdown_connections 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 --2- 172.21.15.63:0/3399093745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a4104d60 0x7f74a4108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 --2- 172.21.15.63:0/3399093745 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a4109480 0x7f74a4111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 --2- 172.21.15.63:0/3399093745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a41043b0 0x7f74a4104790 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 -- 172.21.15.63:0/3399093745 >> 172.21.15.63:0/3399093745 conn(0x7f74a40ffc50 msgr2=0x7f74a4102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 -- 172.21.15.63:0/3399093745 shutdown_connections 2024-04-25T19:05:07.189 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.187+0000 7f74aac00640 1 -- 172.21.15.63:0/3399093745 wait complete. 2024-04-25T19:05:07.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.188+0000 7f74aac00640 1 Processor -- start 2024-04-25T19:05:07.190 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 -- start start 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a41043b0 0x7f74a41a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a419a7d0 0x7f74a419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41143f0 con 0x7f74a4109480 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4114270 con 0x7f74a419a7d0 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74aac00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74a4114570 con 0x7f74a41043b0 2024-04-25T19:05:07.191 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74a3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74a3400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39130/0 (socket says 172.21.15.63:39130) 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74a3400640 1 -- 172.21.15.63:0/1794066981 learned_addr learned my addr 172.21.15.63:0/1794066981 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.189+0000 7f74a3e00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a41043b0 0x7f74a41a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a8c00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a419a7d0 0x7f74a419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 -- 172.21.15.63:0/1794066981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a41043b0 msgr2=0x7f74a41a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a41043b0 0x7f74a41a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 -- 172.21.15.63:0/1794066981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a419a7d0 msgr2=0x7f74a419ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a419a7d0 0x7f74a419ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 -- 172.21.15.63:0/1794066981 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a419b3b0 con 0x7f74a4109480 2024-04-25T19:05:07.192 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a3400640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e9c0 tx=0x7f748c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a0c00640 1 -- 172.21.15.63:0/1794066981 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c00cd80 con 0x7f74a4109480 2024-04-25T19:05:07.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a0c00640 1 -- 172.21.15.63:0/1794066981 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f748c0054d0 con 0x7f74a4109480 2024-04-25T19:05:07.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74a41a7490 con 0x7f74a4109480 2024-04-25T19:05:07.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.190+0000 7f74a0c00640 1 -- 172.21.15.63:0/1794066981 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c010430 con 0x7f74a4109480 2024-04-25T19:05:07.193 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.191+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74a41a7890 con 0x7f74a4109480 2024-04-25T19:05:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 Processor -- start 2024-04-25T19:05:07.195 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 -- start start 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74a0c00640 1 -- 172.21.15.63:0/1794066981 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f748c005020 con 0x7f74a4109480 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c00c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c0ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563f89b66900 con 0x563f89c00820 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563f89b0cef0 con 0x563f89c0ba00 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.194+0000 7f74ada4a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563f89b628c0 con 0x563f89c01140 2024-04-25T19:05:07.196 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c00c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51384/0 (socket says 172.21.15.63:51384) 2024-04-25T19:05:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 -- 172.21.15.63:0/297911396 learned_addr learned my addr 172.21.15.63:0/297911396 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3400640 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c0ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 -- 172.21.15.63:0/297911396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 msgr2=0x563f89c0ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c0ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 -- 172.21.15.63:0/297911396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 msgr2=0x563f89c00c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c00c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a3e00640 1 -- 172.21.15.63:0/297911396 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f89879210 con 0x563f89c01140 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.195+0000 7f74a8c00640 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c00c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f74a3e00640 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f7494032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85581e8ca31d0e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f74a0c00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 0x7f74880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f7499200640 1 -- 172.21.15.63:0/297911396 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494040070 con 0x563f89c01140 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f7499200640 1 -- 172.21.15.63:0/297911396 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7494004030 con 0x563f89c01140 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f7499200640 1 -- 172.21.15.63:0/297911396 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494035040 con 0x563f89c01140 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f74a0c00640 1 -- 172.21.15.63:0/1794066981 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f748c0d0bf0 con 0x7f74a4109480 2024-04-25T19:05:07.199 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.196+0000 7f74a8c00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 0x7f74880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 -- 172.21.15.63:0/297911396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 msgr2=0x563f89c0b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7494009900 tx=0x7f7494032b20 comp rx=0 tx=0).stop 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74a8c00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 0x7f74880a6300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f749c005fd0 tx=0x7f749c009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 -- 172.21.15.63:0/297911396 shutdown_connections 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c0ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c0b4c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 --2- 172.21.15.63:0/297911396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c00c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 -- 172.21.15.63:0/297911396 >> 172.21.15.63:0/297911396 conn(0x563f89bfcd30 msgr2=0x563f89bfd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 -- 172.21.15.63:0/297911396 shutdown_connections 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.197+0000 7f74ada4a740 1 -- 172.21.15.63:0/297911396 wait complete. 2024-04-25T19:05:07.200 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.198+0000 7f74ada4a740 1 -- start start 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.198+0000 7f74ada4a740 1 Processor -- start 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c08960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c0ba00 0x563f89c093e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563f89c0e5d0 con 0x563f89c00820 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563f89b20310 con 0x563f89c01140 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563f89c0e750 con 0x563f89c0ba00 2024-04-25T19:05:07.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74ada4a740 10 client.0 _ll_register_callbacks cb 0x563f89aeecb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c0ba00 0x563f89c093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45666/0 (socket says 172.21.15.63:45666) 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 -- 172.21.15.63:0/3397588697 learned_addr learned my addr 172.21.15.63:0/3397588697 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c08960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 -- 172.21.15.63:0/3397588697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c0ba00 msgr2=0x563f89c093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c0ba00 0x563f89c093e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 -- 172.21.15.63:0/3397588697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 msgr2=0x563f89c08960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c08960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.199+0000 7f74a3400640 1 -- 172.21.15.63:0/3397588697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f89945620 con 0x563f89c01140 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f74a3400640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c6a0 tx=0x7f748c0d4320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcb0405a7be45c85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f7491600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c09c690 con 0x563f89c01140 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f748c0a5330 con 0x563f89c01140 2024-04-25T19:05:07.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7494004030 con 0x563f89c01140 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f748c09c4b0 con 0x563f89c01140 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f74ada4a740 10 client.27314 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563f89c1a850 con 0x563f89c01140 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.200+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7494004030 con 0x563f89c01140 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f748c0988a0 con 0x563f89c01140 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 10 client.27314 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 10 client.27314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 20 client.27314 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c1bcc0 0x563f89c1e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 4 client.27314 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f89b7a8b0 con 0x563f89c1bcc0 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.201+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c1bcc0 0x563f89c1e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.202+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c1bcc0 0x563f89c1e0f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b70eb3917037d75 server_cookie=f577f1ed557c4514 in_seq=0 out_seq=0 2024-04-25T19:05:07.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.202+0000 7f7491600640 10 client.27314 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:07.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f7491600640 1 -- 172.21.15.63:0/3397588697 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f89b7a8b0 con 0x563f89c1bcc0 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f7491600640 10 client.27314 handle_command_reply: tid=0 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 client.27314 shutdown 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 msgr2=0x563f89c08ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f748c00c6a0 tx=0x7f748c0d4320 comp rx=0 tx=0).stop 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 shutdown_connections 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c1bcc0 0x563f89c1e0f0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c01140 0x563f89c08ea0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f7491600640 0 client.27314 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c0ba00 0x563f89c093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3397588697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c00820 0x563f89c08960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 >> 172.21.15.63:0/3397588697 conn(0x563f89bfcd30 msgr2=0x563f89c02ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 shutdown_connections 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 1 -- 172.21.15.63:0/3397588697 wait complete. 2024-04-25T19:05:07.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.203+0000 7f74ada4a740 20 client.27314 trim_cache size 0 max 16384 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 Processor -- start 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 -- start start 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 0x563f89c09030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c239b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b2740 con 0x563f89c01140 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41b0010 con 0x563f89c0ba00 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.220+0000 7f74ada4a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41ade80 con 0x563f89c00820 2024-04-25T19:05:07.222 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:51400/0 (socket says 172.21.15.63:51400) 2024-04-25T19:05:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 -- 172.21.15.63:0/3803686561 learned_addr learned my addr 172.21.15.63:0/3803686561 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a8c00640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3400640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 0x563f89c09030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 -- 172.21.15.63:0/3803686561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 msgr2=0x563f89c239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c239b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.224 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 -- 172.21.15.63:0/3803686561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 msgr2=0x563f89c09030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 0x563f89c09030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 -- 172.21.15.63:0/3803686561 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f89c037d0 con 0x563f89c00820 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3400640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 0x563f89c09030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.221+0000 7f74a3e00640 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x563f89c01dc0 tx=0x7f7494000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19f352a603a13241 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f7493e00640 1 -- 172.21.15.63:0/3803686561 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494055070 con 0x563f89c00820 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f7493e00640 1 -- 172.21.15.63:0/3803686561 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74940314a0 con 0x563f89c00820 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f7493e00640 1 -- 172.21.15.63:0/3803686561 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494040070 con 0x563f89c00820 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 -- 172.21.15.63:0/3803686561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 msgr2=0x563f89c17490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x563f89c01dc0 tx=0x7f7494000bc0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 -- 172.21.15.63:0/3803686561 shutdown_connections 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c0ba00 0x563f89c239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c01140 0x563f89c09030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 --2- 172.21.15.63:0/3803686561 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c00820 0x563f89c17490 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.222+0000 7f74ada4a740 1 -- 172.21.15.63:0/3803686561 >> 172.21.15.63:0/3803686561 conn(0x563f89bfcd30 msgr2=0x563f89c08920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.225 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.223+0000 7f74ada4a740 1 -- 172.21.15.63:0/3803686561 shutdown_connections 2024-04-25T19:05:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.223+0000 7f74ada4a740 1 -- 172.21.15.63:0/3803686561 wait complete. 2024-04-25T19:05:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.223+0000 7f74ada4a740 1 -- start start 2024-04-25T19:05:07.226 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.223+0000 7f74ada4a740 1 Processor -- start 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c13e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c0ba00 0x563f89c10e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41ae4d0 con 0x563f89c0ba00 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41af940 con 0x563f89c00820 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f74a41aeba0 con 0x563f89c01140 2024-04-25T19:05:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74ada4a740 10 client.0 _ll_register_callbacks cb 0x7f748c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a8c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c0ba00 0x563f89c10e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:45686/0 (socket says 172.21.15.63:45686) 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 -- 172.21.15.63:0/2912833560 learned_addr learned my addr 172.21.15.63:0/2912833560 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3400640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c13e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 -- 172.21.15.63:0/2912833560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 msgr2=0x563f89c13e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c13e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 -- 172.21.15.63:0/2912833560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c0ba00 msgr2=0x563f89c10e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c0ba00 0x563f89c10e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3e00640 1 -- 172.21.15.63:0/2912833560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f89c116a0 con 0x563f89c00820 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.224+0000 7f74a3400640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c13e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f74a3e00640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7494009560 tx=0x7f7494005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce2468de385ddfc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f7491600640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7494040070 con 0x563f89c00820 2024-04-25T19:05:07.229 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f7494031850 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74940314a0 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f7494055070 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f74ada4a740 10 client.27320 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.225+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x563f89c8a5c0 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74940314a0 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f7494050710 con 0x563f89c00820 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 10 client.27320 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 10 client.27320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 20 client.27320 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c8a860 0x563f89c8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 4 client.27320 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563f89b74320 con 0x563f89c8a860 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.226+0000 7f74a3400640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c8a860 0x563f89c8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.227+0000 7f74a3400640 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c8a860 0x563f89c8cc50 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77ac1937a8a8b530 server_cookie=776a8d648b290f49 in_seq=0 out_seq=0 2024-04-25T19:05:07.230 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.227+0000 7f7491600640 10 client.27320 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f7491600640 1 -- 172.21.15.63:0/2912833560 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563f89b74320 con 0x563f89c8a860 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f7491600640 10 client.27320 handle_command_reply: tid=0 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 client.27320 shutdown 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 msgr2=0x563f89c13920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7494009560 tx=0x7f7494005fd0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 shutdown_connections 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x563f89c8a860 0x563f89c8cc50 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563f89c0ba00 0x563f89c10e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.256 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563f89c01140 0x563f89c13e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 --2- 172.21.15.63:0/2912833560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f89c00820 0x563f89c13920 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 >> 172.21.15.63:0/2912833560 conn(0x563f89bfcd30 msgr2=0x563f89c091b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f7491600640 0 client.27320 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.254+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 shutdown_connections 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74ada4a740 1 -- 172.21.15.63:0/2912833560 wait complete. 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74ada4a740 20 client.27320 trim_cache size 0 max 16384 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:05:07.257 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:05:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 msgr2=0x7f74880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 0x7f74880a6300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f749c005fd0 tx=0x7f749c009500 comp rx=0 tx=0).stop 2024-04-25T19:05:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 msgr2=0x7f74a41a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.258 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.255+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e9c0 tx=0x7f748c00ee80 comp rx=0 tx=0).stop 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 shutdown_connections 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f74880a3e40 0x7f74880a6300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74a419a7d0 0x7f74a419ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74a4109480 0x7f74a41a0bc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 --2- 172.21.15.63:0/1794066981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f74a41043b0 0x7f74a41a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 >> 172.21.15.63:0/1794066981 conn(0x7f74a40ffc50 msgr2=0x7f74a4101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 shutdown_connections 2024-04-25T19:05:07.259 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:07.256+0000 7f74aac00640 1 -- 172.21.15.63:0/1794066981 wait complete. 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.795+0000 7f7a01200700 1 -- 172.21.15.63:0/444213837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 msgr2=0x7f79fc115f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.795+0000 7f7a01200700 1 --2- 172.21.15.63:0/444213837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc115f80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009f90 tx=0x7f79f400b330 comp rx=0 tx=0).stop 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 -- 172.21.15.63:0/444213837 shutdown_connections 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 --2- 172.21.15.63:0/444213837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc115f80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 --2- 172.21.15.63:0/444213837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 --2- 172.21.15.63:0/444213837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 0x7f79fc10dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 -- 172.21.15.63:0/444213837 >> 172.21.15.63:0/444213837 conn(0x7f79fc076210 msgr2=0x7f79fc107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 -- 172.21.15.63:0/444213837 shutdown_connections 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.796+0000 7f7a01200700 1 -- 172.21.15.63:0/444213837 wait complete. 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 Processor -- start 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 -- start start 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 0x7f79fc19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc1a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1192a0 con 0x7f79fc1092d0 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc119120 con 0x7f79fc10e3b0 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.797+0000 7f7a01200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc119420 con 0x7f79fc109c80 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41628/0 (socket says 172.21.15.63:41628) 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 -- 172.21.15.63:0/1351688642 learned_addr learned my addr 172.21.15.63:0/1351688642 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f3e00700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 0x7f79fc19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79fa800700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc1a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 -- 172.21.15.63:0/1351688642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 msgr2=0x7f79fc19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 0x7f79fc19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 -- 172.21.15.63:0/1351688642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 msgr2=0x7f79fc1a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc1a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79f9e00700 1 -- 172.21.15.63:0/1351688642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc1a4700 con 0x7f79fc1092d0 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.798+0000 7f79fa800700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:07.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f79f9e00700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e9c0 tx=0x7f79ec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f79f2a00700 1 -- 172.21.15.63:0/1351688642 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec004a30 con 0x7f79fc1092d0 2024-04-25T19:05:07.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f79f2a00700 1 -- 172.21.15.63:0/1351688642 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec00cc70 con 0x7f79fc1092d0 2024-04-25T19:05:07.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79fc1a49f0 con 0x7f79fc1092d0 2024-04-25T19:05:07.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f79f2a00700 1 -- 172.21.15.63:0/1351688642 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec005490 con 0x7f79fc1092d0 2024-04-25T19:05:07.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.799+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79fc1ac270 con 0x7f79fc1092d0 2024-04-25T19:05:07.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.803+0000 7f79f2a00700 1 -- 172.21.15.63:0/1351688642 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f79ec004bd0 con 0x7f79fc1092d0 2024-04-25T19:05:07.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.804+0000 7f7a0bbc0b80 1 Processor -- start 2024-04-25T19:05:07.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.804+0000 7f7a0bbc0b80 1 -- start start 2024-04-25T19:05:07.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.804+0000 7f79f2a00700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 0x7f79e00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.804+0000 7f79f2a00700 1 -- 172.21.15.63:0/1351688642 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f79ec0d0910 con 0x7f79fc1092d0 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 0x5561c688df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68908a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79fa800700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 0x7f79e00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5561c67d5f70 con 0x5561c6882d20 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5561c677c810 con 0x5561c6883640 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5561c67d1f30 con 0x5561c688e4b0 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79f3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 0x5561c688df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41640/0 (socket says 172.21.15.63:41640) 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79f3e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 0x5561c688df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47222/0 (socket says 172.21.15.63:47222) 2024-04-25T19:05:07.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.805+0000 7f79f9e00700 1 -- 172.21.15.63:0/679251993 learned_addr learned my addr 172.21.15.63:0/679251993 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 -- 172.21.15.63:0/679251993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 msgr2=0x5561c68908a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68908a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 -- 172.21.15.63:0/679251993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 msgr2=0x5561c688df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 0x5561c688df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 -- 172.21.15.63:0/679251993 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c6772760 con 0x5561c6882d20 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79fa800700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 0x7f79e00a6270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009da0 tx=0x7f79f40092e0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79f9e00700 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0a4fb0 tx=0x7f79ec09c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3356c8d84018499 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79eb400700 1 -- 172.21.15.63:0/679251993 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec09c870 con 0x5561c6882d20 2024-04-25T19:05:07.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79eb400700 1 -- 172.21.15.63:0/679251993 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec0a0c20 con 0x5561c6882d20 2024-04-25T19:05:07.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.806+0000 7f79eb400700 1 -- 172.21.15.63:0/679251993 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0870c0 con 0x5561c6882d20 2024-04-25T19:05:07.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/679251993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 msgr2=0x5561c6883100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0a4fb0 tx=0x7f79ec09c1d0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/679251993 shutdown_connections 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68908a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6882d20 0x5561c6883100 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/679251993 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6883640 0x5561c688df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.807+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/679251993 >> 172.21.15.63:0/679251993 conn(0x5561c67b2030 msgr2=0x5561c687f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.808+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/679251993 shutdown_connections 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.808+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/679251993 wait complete. 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.808+0000 7f7a0bbc0b80 1 -- start start 2024-04-25T19:05:07.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.808+0000 7f7a0bbc0b80 1 Processor -- start 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c688af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c688b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5561c6890fa0 con 0x5561c688e4b0 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5561c678f940 con 0x5561c6882d20 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5561c6891120 con 0x5561c6883640 2024-04-25T19:05:07.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f7a0bbc0b80 10 client.0 _ll_register_callbacks cb 0x5561c676e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f79f9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:47236/0 (socket says 172.21.15.63:47236) 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f79f9e00700 1 -- 172.21.15.63:0/2624086627 learned_addr learned my addr 172.21.15.63:0/2624086627 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.809+0000 7f79fa800700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c688b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f3e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c688af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 -- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 msgr2=0x5561c688af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c688af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 -- 172.21.15.63:0/2624086627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 msgr2=0x5561c688b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c688b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 -- 172.21.15.63:0/2624086627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c67239b0 con 0x5561c6882d20 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79f9e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f79ec000c00 tx=0x7f79ec0a54e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9163db6f20299bc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79dfe00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec066060 con 0x5561c6882d20 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f79ec09cad0 con 0x5561c6882d20 2024-04-25T19:05:07.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.810+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79ec0a0c20 con 0x5561c6882d20 2024-04-25T19:05:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f79ec0bb070 con 0x5561c6882d20 2024-04-25T19:05:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 10 client.27332 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5561c676cbd0 con 0x5561c6882d20 2024-04-25T19:05:07.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79ec0a0c20 con 0x5561c6882d20 2024-04-25T19:05:07.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f79ec0a5050 con 0x5561c6882d20 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 10 client.27332 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 10 client.27332 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 20 client.27332 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c689db20 0x5561c689ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 4 client.27332 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.811+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561c67e3760 con 0x5561c689db20 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.812+0000 7f79f3e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c689db20 0x5561c689ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.812+0000 7f79f3e00700 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c689db20 0x5561c689ff10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75a552c11b246fb5 server_cookie=9887e9efa401d761 in_seq=0 out_seq=0 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.812+0000 7f79dfe00700 10 client.27332 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f79dfe00700 1 -- 172.21.15.63:0/2624086627 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561c67e3760 con 0x5561c689db20 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f79dfe00700 10 client.27332 handle_command_reply: tid=0 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 client.27332 shutdown 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 msgr2=0x5561c688dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f79ec000c00 tx=0x7f79ec0a54e0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 shutdown_connections 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c689db20 0x5561c689ff10 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c688b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688dc00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/2624086627 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c688af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f79dfe00700 0 client.27332 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.813+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 >> 172.21.15.63:0/2624086627 conn(0x5561c67b2030 msgr2=0x5561c687ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.814+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 shutdown_connections 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.814+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/2624086627 wait complete. 2024-04-25T19:05:07.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.814+0000 7f7a0bbc0b80 20 client.27332 trim_cache size 0 max 16384 2024-04-25T19:05:07.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.833+0000 7f7a0bbc0b80 1 Processor -- start 2024-04-25T19:05:07.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.833+0000 7f7a0bbc0b80 1 -- start start 2024-04-25T19:05:07.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b63c0 con 0x5561c6883640 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b3c50 con 0x5561c6882d20 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b1ac0 con 0x5561c688e4b0 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f79f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f79fa800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f79f3e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41654/0 (socket says 172.21.15.63:41654) 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f79f3e00700 1 -- 172.21.15.63:0/3711670031 learned_addr learned my addr 172.21.15.63:0/3711670031 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.834+0000 7f79f9e00700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 -- 172.21.15.63:0/3711670031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 msgr2=0x5561c688a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 -- 172.21.15.63:0/3711670031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 msgr2=0x5561c688ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 -- 172.21.15.63:0/3711670031 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561c68804a0 con 0x5561c688e4b0 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79f9e00700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:07.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79f3e00700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:07.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79fa800700 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68a2f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x5561c68846f0 tx=0x7f79f401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4447dec683fd616 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79eaa00700 1 -- 172.21.15.63:0/3711670031 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f402e070 con 0x5561c688e4b0 2024-04-25T19:05:07.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79eaa00700 1 -- 172.21.15.63:0/3711670031 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f4007c90 con 0x5561c688e4b0 2024-04-25T19:05:07.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.835+0000 7f79eaa00700 1 -- 172.21.15.63:0/3711670031 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4033cc0 con 0x5561c688e4b0 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/3711670031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 msgr2=0x5561c68a2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68a2f40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x5561c68846f0 tx=0x7f79f401f000 comp rx=0 tx=0).stop 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/3711670031 shutdown_connections 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c688e4b0 0x5561c68a2f40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c688a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/3711670031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c6883640 0x5561c688ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/3711670031 >> 172.21.15.63:0/3711670031 conn(0x5561c67b2030 msgr2=0x5561c688c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/3711670031 shutdown_connections 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.836+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/3711670031 wait complete. 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.837+0000 7f7a0bbc0b80 1 -- start start 2024-04-25T19:05:07.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.837+0000 7f7a0bbc0b80 1 Processor -- start 2024-04-25T19:05:07.841 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c68fd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c68fdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c68f7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b2110 con 0x5561c688e4b0 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b3580 con 0x5561c6882d20 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc1b27e0 con 0x5561c6883640 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f7a0bbc0b80 10 client.0 _ll_register_callbacks cb 0x5561c676e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f79fa800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c68f7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f79f3e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c68fdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f79fa800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c68f7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41656/0 (socket says 172.21.15.63:41656) 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f79fa800700 1 -- 172.21.15.63:0/834486378 learned_addr learned my addr 172.21.15.63:0/834486378 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.838+0000 7f79f9e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c68fd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 -- 172.21.15.63:0/834486378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 msgr2=0x5561c68fd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c68fd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 -- 172.21.15.63:0/834486378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 msgr2=0x5561c68f7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c68f7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 -- 172.21.15.63:0/834486378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561c68f8570 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79f3e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c68fdf10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004790 tx=0x7f79e40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70f3a48e53a438cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79dfe00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4004a30 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f4007c90 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f79e4004bd0 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f79e402f050 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f7a0bbc0b80 10 client.27346 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.839+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5561c6905290 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79f4007c90 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f79e40203f0 con 0x5561c6883640 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 10 client.27346 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 10 client.27346 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T19:05:07.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 20 client.27346 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c69059c0 0x5561c6907de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 4 client.27346 mds_command: new command op to 24520 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.i.json"}] 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 --> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.i.json"}) -- 0x5561c67dd800 con 0x5561c69059c0 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.840+0000 7f79f9e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c69059c0 0x5561c6907de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.841+0000 7f79f9e00700 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c69059c0 0x5561c6907de0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9df45241aeec7f7 server_cookie=1cb639280bec5caa in_seq=0 out_seq=0 2024-04-25T19:05:07.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.841+0000 7f79dfe00700 10 client.27346 ms_handle_connect on v2:172.21.15.114:6838/2417088150 2024-04-25T19:05:07.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f79dfe00700 1 -- 172.21.15.63:0/834486378 <== mds.1 v2:172.21.15.114:6838/2417088150 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x5561c67dd800 con 0x5561c69059c0 2024-04-25T19:05:07.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f79dfe00700 10 client.27346 handle_command_reply: tid=0 2024-04-25T19:05:07.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 client.27346 shutdown 2024-04-25T19:05:07.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 msgr2=0x5561c68fdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c68fdf10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f79e4004790 tx=0x7f79e40107a0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 shutdown_connections 2024-04-25T19:05:07.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:6838/2417088150,v1:172.21.15.114:6839/2417088150] conn(0x5561c69059c0 0x5561c6907de0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5561c688e4b0 0x5561c68f7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5561c6882d20 0x5561c68fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 --2- 172.21.15.63:0/834486378 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5561c6883640 0x5561c68fdf10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f79dfe00700 0 client.27346 ms_handle_reset on v2:172.21.15.114:6838/2417088150 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 >> 172.21.15.63:0/834486378 conn(0x5561c67b2030 msgr2=0x5561c688c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 shutdown_connections 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.864+0000 7f7a0bbc0b80 1 -- 172.21.15.63:0/834486378 wait complete. 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.865+0000 7f7a0bbc0b80 20 client.27346 trim_cache size 0 max 16384 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stdout: "path": "/var/run/ceph/ops-e43e1e63-1-mds.i.json" 2024-04-25T19:05:07.867 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.865+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 msgr2=0x7f79e00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.865+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 0x7f79e00a6270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009da0 tx=0x7f79f40092e0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 msgr2=0x7f79fc19f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e9c0 tx=0x7f79ec00ecd0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 shutdown_connections 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f79e00a3db0 0x7f79e00a6270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79fc10e3b0 0x7f79fc1a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79fc1092d0 0x7f79fc19f6f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 --2- 172.21.15.63:0/1351688642 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f79fc109c80 0x7f79fc19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:07.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 >> 172.21.15.63:0/1351688642 conn(0x7f79fc076210 msgr2=0x7f79fc076750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:07.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 shutdown_connections 2024-04-25T19:05:07.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:07.866+0000 7f7a01200700 1 -- 172.21.15.63:0/1351688642 wait complete. 2024-04-25T19:05:08.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.85 MB/sec execute 186 sec latency 61069.954 ms 2024-04-25T19:05:08.606 DEBUG:tasks.quiescer.fs.[cephfs]:Dumping ops on rank 0 (b) to a remote file /var/run/ceph/1/ops-e43e1e63-1-mds.b.json 2024-04-25T19:05:08.606 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph tell mds.1:0 ops --flags=locks --path=/var/run/ceph/ops-e43e1e63-1-mds.b.json 2024-04-25T19:05:08.733 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:08 smithi063 ceph-mon[28006]: pgmap v588: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 22 op/s 2024-04-25T19:05:08.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:08 smithi114 ceph-mon[32988]: pgmap v588: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 22 op/s 2024-04-25T19:05:08.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:08 smithi070 ceph-mon[32820]: pgmap v588: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 22 op/s 2024-04-25T19:05:09.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.76 MB/sec execute 187 sec latency 62070.038 ms 2024-04-25T19:05:09.123 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:05:10.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.67 MB/sec execute 188 sec latency 63070.157 ms 2024-04-25T19:05:10.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:10 smithi114 ceph-mon[32988]: pgmap v589: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:10.856 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:10 smithi063 ceph-mon[28006]: pgmap v589: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:10.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:10 smithi070 ceph-mon[32820]: pgmap v589: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:11.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.58 MB/sec execute 189 sec latency 64070.259 ms 2024-04-25T19:05:11.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.396+0000 7f0bb4200700 1 -- 172.21.15.63:0/643654867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06a750 msgr2=0x7f0bac06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.396+0000 7f0bb4200700 1 --2- 172.21.15.63:0/643654867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06a750 0x7f0bac06abb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0b980098f0 tx=0x7f0b98009c00 comp rx=0 tx=0).stop 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 -- 172.21.15.63:0/643654867 shutdown_connections 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 --2- 172.21.15.63:0/643654867 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 0x7f0bac108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 --2- 172.21.15.63:0/643654867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06b0f0 0x7f0bac10f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 --2- 172.21.15.63:0/643654867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06a750 0x7f0bac06abb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 -- 172.21.15.63:0/643654867 >> 172.21.15.63:0/643654867 conn(0x7f0bac0788f0 msgr2=0x7f0bac078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 -- 172.21.15.63:0/643654867 shutdown_connections 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.397+0000 7f0bb4200700 1 -- 172.21.15.63:0/643654867 wait complete. 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.398+0000 7f0bb4200700 1 Processor -- start 2024-04-25T19:05:11.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.398+0000 7f0bb4200700 1 -- start start 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.398+0000 7f0bb4200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 0x7f0bac19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.398+0000 7f0bb4200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 0x7f0bac19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.398+0000 7f0bb4200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 0x7f0bac19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb4200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac112c20 con 0x7f0bac06a750 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb4200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac112aa0 con 0x7f0bac06b0f0 2024-04-25T19:05:11.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb4200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac112da0 con 0x7f0bac108a30 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 0x7f0bac19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 0x7f0bac19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 0x7f0bac19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41668/0 (socket says 172.21.15.63:41668) 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb1800700 1 -- 172.21.15.63:0/2971923344 learned_addr learned my addr 172.21.15.63:0/2971923344 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb2200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 0x7f0bac19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 -- 172.21.15.63:0/2971923344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 msgr2=0x7f0bac19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 0x7f0bac19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 -- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 msgr2=0x7f0bac19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 0x7f0bac19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.399+0000 7f0bb0e00700 1 -- 172.21.15.63:0/2971923344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac1a03a0 con 0x7f0bac06b0f0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 0x7f0bac19b8d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c010 tx=0x7f0b9800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0baa000700 1 -- 172.21.15.63:0/2971923344 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b980050c0 con 0x7f0bac06b0f0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0baa000700 1 -- 172.21.15.63:0/2971923344 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b9800b560 con 0x7f0bac06b0f0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0baa000700 1 -- 172.21.15.63:0/2971923344 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98003a00 con 0x7f0bac06b0f0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bac1a0630 con 0x7f0bac06b0f0 2024-04-25T19:05:11.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.400+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bac1a7f10 con 0x7f0bac06b0f0 2024-04-25T19:05:11.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.405+0000 7f0baa000700 1 -- 172.21.15.63:0/2971923344 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0b9800b710 con 0x7f0bac06b0f0 2024-04-25T19:05:11.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.406+0000 7f0bbebdfb80 1 Processor -- start 2024-04-25T19:05:11.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 -- start start 2024-04-25T19:05:11.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 0x55c1f341dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 0x55c1f3428b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f342b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1f3370f70 con 0x55c1f34290d0 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1f3317810 con 0x55c1f341e220 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1f336cf30 con 0x55c1f341d900 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bb2200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 0x55c1f341dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bb2200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 0x55c1f341dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44326/0 (socket says 172.21.15.63:44326) 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.407+0000 7f0bb2200700 1 -- 172.21.15.63:0/3399497611 learned_addr learned my addr 172.21.15.63:0/3399497611 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f342b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0baa000700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 0x7f0b940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 0x55c1f3428b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0baa000700 1 -- 172.21.15.63:0/2971923344 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f0b980d1620 con 0x7f0bac06b0f0 2024-04-25T19:05:11.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 -- 172.21.15.63:0/3399497611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 msgr2=0x55c1f341dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 0x55c1f341dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 -- 172.21.15.63:0/3399497611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 msgr2=0x55c1f342b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f342b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb1800700 1 -- 172.21.15.63:0/3399497611 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1f330d760 con 0x55c1f341e220 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.408+0000 7f0bb2200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 0x7f0b940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.409+0000 7f0bb1800700 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 0x55c1f3428b90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009cc0 tx=0x7f0ba0011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96940b07489cfe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.409+0000 7f0b9ea00700 1 -- 172.21.15.63:0/3399497611 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0011d50 con 0x55c1f341e220 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.409+0000 7f0b9ea00700 1 -- 172.21.15.63:0/3399497611 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba0004500 con 0x55c1f341e220 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.409+0000 7f0b9ea00700 1 -- 172.21.15.63:0/3399497611 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0005280 con 0x55c1f341e220 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/3399497611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 msgr2=0x55c1f3428b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bb2200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 0x7f0b940a6270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006220 tx=0x7f0ba4009040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 0x55c1f3428b90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009cc0 tx=0x7f0ba0011440 comp rx=0 tx=0).stop 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/3399497611 shutdown_connections 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341d900 0x55c1f341dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f342b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/3399497611 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341e220 0x55c1f3428b90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/3399497611 >> 172.21.15.63:0/3399497611 conn(0x55c1f334c680 msgr2=0x55c1f3419db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/3399497611 shutdown_connections 2024-04-25T19:05:11.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.410+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/3399497611 wait complete. 2024-04-25T19:05:11.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.411+0000 7f0bbebdfb80 1 -- start start 2024-04-25T19:05:11.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.411+0000 7f0bbebdfb80 1 Processor -- start 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3425ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f3426000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c1f342bbf0 con 0x55c1f341d900 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c1f332a940 con 0x55c1f34290d0 2024-04-25T19:05:11.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c1f342bd70 con 0x55c1f341e220 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bbebdfb80 10 client.0 _ll_register_callbacks cb 0x55c1f3309800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41688/0 (socket says 172.21.15.63:41688) 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb1800700 1 -- 172.21.15.63:0/1226676117 learned_addr learned my addr 172.21.15.63:0/1226676117 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb2200700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f3426000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3425ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb1800700 1 -- 172.21.15.63:0/1226676117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 msgr2=0x55c1f3425ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.412+0000 7f0bb1800700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3425ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0bb1800700 1 -- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 msgr2=0x55c1f3426000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0bb1800700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f3426000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0bb1800700 1 -- 172.21.15.63:0/1226676117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1f32be9b0 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0bb1800700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0011fa0 tx=0x7f0ba00040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62835fe9b438eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0b93400700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0004750 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f0ba00048f0 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.413+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ba0004500 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0ba0030050 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 10 client.17118 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55c1f3307bd0 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba0004500 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f0ba0018070 con 0x55c1f341d900 2024-04-25T19:05:11.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 10 client.17118 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 20 client.17118 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34387e0 0x55c1f343abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 4 client.17118 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.414+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1f337e760 con 0x55c1f34387e0 2024-04-25T19:05:11.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.415+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34387e0 0x55c1f343abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.415+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34387e0 0x55c1f343abd0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=144f37746846fba9 server_cookie=1f2f55f3bcab162c in_seq=0 out_seq=0 2024-04-25T19:05:11.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.415+0000 7f0b93400700 10 client.17118 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:11.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0b93400700 1 -- 172.21.15.63:0/1226676117 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1f337e760 con 0x55c1f34387e0 2024-04-25T19:05:11.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0b93400700 10 client.17118 handle_command_reply: tid=0 2024-04-25T19:05:11.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0bbebdfb80 1 client.17118 shutdown 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 msgr2=0x55c1f3428750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0011fa0 tx=0x7f0ba00040d0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 shutdown_connections 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.416+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34387e0 0x55c1f343abd0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f3426000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f3428750 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0b93400700 0 client.17118 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/1226676117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3425ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 >> 172.21.15.63:0/1226676117 conn(0x55c1f334c680 msgr2=0x55c1f34199a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 shutdown_connections 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/1226676117 wait complete. 2024-04-25T19:05:11.421 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.417+0000 7f0bbebdfb80 20 client.17118 trim_cache size 0 max 16384 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 Processor -- start 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 -- start start 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 0x55c1f3425f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3434af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1b1e20 con 0x55c1f34290d0 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1af6b0 con 0x55c1f341d900 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.437+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1ad520 con 0x55c1f341e220 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41700/0 (socket says 172.21.15.63:41700) 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 -- 172.21.15.63:0/699005977 learned_addr learned my addr 172.21.15.63:0/699005977 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb1800700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 0x55c1f3425f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3434af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 -- 172.21.15.63:0/699005977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 msgr2=0x55c1f3434af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3434af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 -- 172.21.15.63:0/699005977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 msgr2=0x55c1f3425f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 0x55c1f3425f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb2200700 1 -- 172.21.15.63:0/699005977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1f3427020 con 0x55c1f34290d0 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.438+0000 7f0bb1800700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 0x55c1f3425f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.439+0000 7f0bb2200700 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40179e0 tx=0x7f0ba4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b715e3ef200df9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.439+0000 7f0b9e000700 1 -- 172.21.15.63:0/699005977 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4004bb0 con 0x55c1f34290d0 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.439+0000 7f0b9e000700 1 -- 172.21.15.63:0/699005977 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba4004d50 con 0x55c1f34290d0 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.439+0000 7f0b9e000700 1 -- 172.21.15.63:0/699005977 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4005440 con 0x55c1f34290d0 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/699005977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 msgr2=0x55c1f3427a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40179e0 tx=0x7f0ba4004b40 comp rx=0 tx=0).stop 2024-04-25T19:05:11.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/699005977 shutdown_connections 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f34290d0 0x55c1f3427a00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f341d900 0x55c1f3425f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/699005977 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f3434af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/699005977 >> 172.21.15.63:0/699005977 conn(0x55c1f334c680 msgr2=0x55c1f34272f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/699005977 shutdown_connections 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/699005977 wait complete. 2024-04-25T19:05:11.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.440+0000 7f0bbebdfb80 1 -- start start 2024-04-25T19:05:11.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 Processor -- start 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f342bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f342c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1adb70 con 0x55c1f341d900 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1aefe0 con 0x55c1f34290d0 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac1ae240 con 0x55c1f341e220 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bbebdfb80 10 client.0 _ll_register_callbacks cb 0x55c1f3309800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bb1800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:41714/0 (socket says 172.21.15.63:41714) 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bb1800700 1 -- 172.21.15.63:0/2005120628 learned_addr learned my addr 172.21.15.63:0/2005120628 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.441+0000 7f0bb2200700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f342c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f342bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 -- 172.21.15.63:0/2005120628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 msgr2=0x55c1f342bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f342bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 -- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 msgr2=0x55c1f342c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f342c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 -- 172.21.15.63:0/2005120628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1f342cd40 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb2200700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f342c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0bb1800700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00052d0 tx=0x7f0ba0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed363e6353d5998 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.442+0000 7f0b93400700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0039060 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f0ba0003890 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ba4004d50 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f0ba0015070 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 10 client.17124 fetch_fsmap learned FSMap version 58 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55c1f349fb50 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba4004d50 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 58) ==== 8338+0+0 (secure 0 0 0) 0x7f0ba0018040 con 0x55c1f341d900 2024-04-25T19:05:11.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 10 client.17124 fetch_fsmap finished waiting for FSMap version 58 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.443+0000 7f0bbebdfb80 20 client.17124 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34a0250 0x55c1f34a2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0bbebdfb80 4 client.17124 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.b.json"}] 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/var/run/ceph/ops-e43e1e63-1-mds.b.json"}) -- 0x55c1f3378800 con 0x55c1f34a0250 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34a0250 0x55c1f34a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0bb0e00700 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34a0250 0x55c1f34a2670 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ce4e841dd18f6f server_cookie=62e530245de40417 in_seq=0 out_seq=0 2024-04-25T19:05:11.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.444+0000 7f0b93400700 10 client.17124 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:11.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0b93400700 1 -- 172.21.15.63:0/2005120628 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+58 (crc 0 0 0) 0x55c1f3378800 con 0x55c1f34a0250 2024-04-25T19:05:11.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0b93400700 10 client.17124 handle_command_reply: tid=0 2024-04-25T19:05:11.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 client.17124 shutdown 2024-04-25T19:05:11.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 msgr2=0x55c1f34313d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00052d0 tx=0x7f0ba0011e60 comp rx=0 tx=0).stop 2024-04-25T19:05:11.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 shutdown_connections 2024-04-25T19:05:11.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c1f34a0250 0x55c1f34a2670 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c1f34290d0 0x55c1f342c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c1f341d900 0x55c1f34313d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 --2- 172.21.15.63:0/2005120628 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c1f341e220 0x55c1f342bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 >> 172.21.15.63:0/2005120628 conn(0x55c1f334c680 msgr2=0x55c1f3418ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0b93400700 0 client.17124 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 shutdown_connections 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.467+0000 7f0bbebdfb80 1 -- 172.21.15.63:0/2005120628 wait complete. 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bbebdfb80 20 client.17124 trim_cache size 0 max 16384 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stdout: "path": "/var/run/ceph/ops-e43e1e63-1-mds.b.json" 2024-04-25T19:05:11.470 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 msgr2=0x7f0b940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 0x7f0b940a6270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4006220 tx=0x7f0ba4009040 comp rx=0 tx=0).stop 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 msgr2=0x7f0bac19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 0x7f0bac19b8d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c010 tx=0x7f0b9800fa00 comp rx=0 tx=0).stop 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 shutdown_connections 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0b940a3db0 0x7f0b940a6270 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0bac108a30 0x7f0bac19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0bac06a750 0x7f0bac19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.468+0000 7f0bb4200700 1 --2- 172.21.15.63:0/2971923344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0bac06b0f0 0x7f0bac19b8d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.469+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 >> 172.21.15.63:0/2971923344 conn(0x7f0bac0788f0 msgr2=0x7f0bac10dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.469+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 shutdown_connections 2024-04-25T19:05:11.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:11.469+0000 7f0bb4200700 1 -- 172.21.15.63:0/2971923344 wait complete. 2024-04-25T19:05:12.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.49 MB/sec execute 190 sec latency 65070.364 ms 2024-04-25T19:05:12.396 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-04-25T19:05:12.415 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.7juDeLTNpC 2024-04-25T19:05:12.415 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.c.json /tmp/tmp.7juDeLTNpC 2024-04-25T19:05:12.485 INFO:teuthology.orchestra.run.smithi114.stderr:cp: cannot stat '/var/run/ceph/1/ops-e43e1e63-1-mds.c.json': No such file or directory 2024-04-25T19:05:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:05:12.486 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't pull ops dump at '/var/run/ceph/1/ops-e43e1e63-1-mds.c.json' on rank 2 (c), error: Command failed on smithi114 with status 1: 'sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.c.json /tmp/tmp.7juDeLTNpC' 2024-04-25T19:05:12.487 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-04-25T19:05:12.543 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.PaqKptRWFY 2024-04-25T19:05:12.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.i.json /tmp/tmp.PaqKptRWFY 2024-04-25T19:05:12.608 INFO:teuthology.orchestra.run.smithi114.stderr:cp: cannot stat '/var/run/ceph/1/ops-e43e1e63-1-mds.i.json': No such file or directory 2024-04-25T19:05:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:05:12.610 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't pull ops dump at '/var/run/ceph/1/ops-e43e1e63-1-mds.i.json' on rank 1 (i), error: Command failed on smithi114 with status 1: 'sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.i.json /tmp/tmp.PaqKptRWFY' 2024-04-25T19:05:12.610 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-25T19:05:12.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:12 smithi063 ceph-mon[28006]: pgmap v590: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:12.629 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.YiynNtz2w0 2024-04-25T19:05:12.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.b.json /tmp/tmp.YiynNtz2w0 2024-04-25T19:05:12.698 INFO:teuthology.orchestra.run.smithi070.stderr:cp: cannot stat '/var/run/ceph/1/ops-e43e1e63-1-mds.b.json': No such file or directory 2024-04-25T19:05:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:05:12.700 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't pull ops dump at '/var/run/ceph/1/ops-e43e1e63-1-mds.b.json' on rank 0 (b), error: Command failed on smithi070 with status 1: 'sudo cp /var/run/ceph/1/ops-e43e1e63-1-mds.b.json /tmp/tmp.YiynNtz2w0' 2024-04-25T19:05:12.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:05:12.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:12 smithi114 ceph-mon[32988]: pgmap v590: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:12 smithi070 ceph-mon[32820]: pgmap v590: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:13.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.40 MB/sec execute 191 sec latency 66070.438 ms 2024-04-25T19:05:13.170 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:05:14.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 109875 16.32 MB/sec execute 192 sec latency 67070.532 ms 2024-04-25T19:05:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:14 smithi114 ceph-mon[32988]: pgmap v591: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:14.864 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:14 smithi063 ceph-mon[28006]: pgmap v591: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:14.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:14 smithi070 ceph-mon[32820]: pgmap v591: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 48 KiB/s wr, 26 op/s 2024-04-25T19:05:15.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 110171 16.29 MB/sec execute 193 sec latency 67445.534 ms 2024-04-25T19:05:15.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 -- 172.21.15.63:0/438508327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 msgr2=0x7f1d3410ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:15.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/438508327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3410ef00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009990 tx=0x7f1d20009ca0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 -- 172.21.15.63:0/438508327 shutdown_connections 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/438508327 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d34075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/438508327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d3410f440 0x7f1d34111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/438508327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3410ef00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.467+0000 7f1d3aa00700 1 -- 172.21.15.63:0/438508327 >> 172.21.15.63:0/438508327 conn(0x7f1d34079ba0 msgr2=0x7f1d34079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.468+0000 7f1d3aa00700 1 -- 172.21.15.63:0/438508327 shutdown_connections 2024-04-25T19:05:15.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.468+0000 7f1d3aa00700 1 -- 172.21.15.63:0/438508327 wait complete. 2024-04-25T19:05:15.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 Processor -- start 2024-04-25T19:05:15.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 -- start start 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3419b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d3410f440 0x7f1d3419fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1d34114e50 con 0x7f1d3410f440 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1d34114cd0 con 0x7f1d34075590 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.470+0000 7f1d3aa00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f1d34114fd0 con 0x7f1d34074c70 2024-04-25T19:05:15.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:15.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:44386/0 (socket says 172.21.15.63:44386) 2024-04-25T19:05:15.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 -- 172.21.15.63:0/278239753 learned_addr learned my addr 172.21.15.63:0/278239753 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33400700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3419b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 -- 172.21.15.63:0/278239753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 msgr2=0x7f1d3419b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3419b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 -- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d3410f440 msgr2=0x7f1d3419fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.471+0000 7f1d33e00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d3410f440 0x7f1d3419fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.472+0000 7f1d33e00700 1 -- 172.21.15.63:0/278239753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d341a0370 con 0x7f1d34074c70 2024-04-25T19:05:15.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.472+0000 7f1d33e00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e9a0 tx=0x7f1d2400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:15.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.473+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2400cbd0 con 0x7f1d34074c70 2024-04-25T19:05:15.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.473+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d2400ee90 con 0x7f1d34074c70 2024-04-25T19:05:15.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.473+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d24018680 con 0x7f1d34074c70 2024-04-25T19:05:15.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.473+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d341a0660 con 0x7f1d34074c70 2024-04-25T19:05:15.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.473+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d341a7ee0 con 0x7f1d34074c70 2024-04-25T19:05:15.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.475+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d34069730 con 0x7f1d34074c70 2024-04-25T19:05:15.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.478+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f1d24010430 con 0x7f1d34074c70 2024-04-25T19:05:15.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.479+0000 7f1d30c00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 0x7f1d180a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:15.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.479+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f1d240d0bd0 con 0x7f1d34074c70 2024-04-25T19:05:15.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.481+0000 7f1d33400700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 0x7f1d180a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:15.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.482+0000 7f1d33400700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 0x7f1d180a63c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1d341111e0 tx=0x7f1d2000b260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:15.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.483+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d2409d360 con 0x7f1d34074c70 2024-04-25T19:05:15.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:15 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:15.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:15 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:15.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.922+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d3402df80 con 0x7f1d34074c70 2024-04-25T19:05:15.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.923+0000 7f1d30c00700 1 -- 172.21.15.63:0/278239753 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+11865 (secure 0 0 0) 0x7f1d2409cc60 con 0x7f1d34074c70 2024-04-25T19:05:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:05:15.925 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":58,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:04:46.191966+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24508,24520,24533]},"id":1}]} 2024-04-25T19:05:15.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.927+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 msgr2=0x7f1d180a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:15.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.927+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 0x7f1d180a63c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1d341111e0 tx=0x7f1d2000b260 comp rx=0 tx=0).stop 2024-04-25T19:05:15.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.927+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 msgr2=0x7f1d3419b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:15.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.927+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400e9a0 tx=0x7f1d2400ecb0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 shutdown_connections 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f1d180a3f00 0x7f1d180a63c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f1d34074c70 0x7f1d3419b360 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d3410f440 0x7f1d3419fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 --2- 172.21.15.63:0/278239753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1d34075590 0x7f1d3419b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 >> 172.21.15.63:0/278239753 conn(0x7f1d34079ba0 msgr2=0x7f1d340ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 shutdown_connections 2024-04-25T19:05:15.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:15.928+0000 7f1d3aa00700 1 -- 172.21.15.63:0/278239753 wait complete. 2024-04-25T19:05:15.934 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 58 2024-04-25T19:05:15.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:15 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:05:16.112 INFO:tasks.workunit.client.0.smithi063.stdout: 10 110698 16.31 MB/sec execute 194 sec latency 92.416 ms 2024-04-25T19:05:16.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: pgmap v592: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:16.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/278239753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:16.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.784 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --set-id e43e1e63 --await-for 42.0' 2024-04-25T19:05:16.784 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --set-id e43e1e63 --await-for 42.0 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: pgmap v592: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/278239753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: pgmap v592: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 30 op/s 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/278239753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:16.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:17.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 111241 16.39 MB/sec execute 195 sec latency 82.462 ms 2024-04-25T19:05:17.297 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:05:18.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 111847 16.47 MB/sec execute 196 sec latency 97.201 ms 2024-04-25T19:05:18.773 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:18 smithi063 ceph-mon[28006]: pgmap v593: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-25T19:05:18.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:18 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:18.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:18 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:18.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:18 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:18 smithi114 ceph-mon[32988]: pgmap v593: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-25T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:18 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:18 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:18 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:18 smithi070 ceph-mon[32820]: pgmap v593: 97 pgs: 97 active+clean; 923 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-25T19:05:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:18 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:05:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:18 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:05:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:18 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:05:19.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 112511 16.63 MB/sec execute 197 sec latency 73.081 ms 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 -- 172.21.15.63:0/3350792837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d006ac90 msgr2=0x7fa9d006b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 --2- 172.21.15.63:0/3350792837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d006ac90 0x7fa9d006b0f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc01a9a0 tx=0x7fa9bc01acb0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 -- 172.21.15.63:0/3350792837 shutdown_connections 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 --2- 172.21.15.63:0/3350792837 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9d0105480 0x7fa9d0105910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 --2- 172.21.15.63:0/3350792837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d0107970 0x7fa9d006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 --2- 172.21.15.63:0/3350792837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d006ac90 0x7fa9d006b0f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 -- 172.21.15.63:0/3350792837 >> 172.21.15.63:0/3350792837 conn(0x7fa9d00788f0 msgr2=0x7fa9d0078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 -- 172.21.15.63:0/3350792837 shutdown_connections 2024-04-25T19:05:19.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.725+0000 7fa9d8600700 1 -- 172.21.15.63:0/3350792837 wait complete. 2024-04-25T19:05:19.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.727+0000 7fa9d8600700 1 Processor -- start 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 -- start start 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d006ac90 0x7fa9d0111ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9d0105480 0x7fa9d0112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0107970 0x7fa9d0115790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010a780 con 0x7fa9d006ac90 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010a600 con 0x7fa9d0107970 2024-04-25T19:05:19.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d8600700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010a900 con 0x7fa9d0105480 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0107970 0x7fa9d0115790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d5200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9d0105480 0x7fa9d0112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0107970 0x7fa9d0115790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43272/0 (socket says 172.21.15.63:43272) 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.729+0000 7fa9d6600700 1 -- 172.21.15.63:0/2550723464 learned_addr learned my addr 172.21.15.63:0/2550723464 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d006ac90 0x7fa9d0111ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5200700 1 -- 172.21.15.63:0/2550723464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0107970 msgr2=0x7fa9d0115790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5200700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9d0107970 0x7fa9d0115790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5200700 1 -- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d006ac90 msgr2=0x7fa9d0111ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5200700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d006ac90 0x7fa9d0111ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.730+0000 7fa9d5200700 1 -- 172.21.15.63:0/2550723464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d0115ed0 con 0x7fa9d0105480 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.731+0000 7fa9d5200700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa9d0105480 0x7fa9d0112410 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc01c2f0 tx=0x7fa9bc0299a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.731+0000 7fa9ce000700 1 -- 172.21.15.63:0/2550723464 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc004020 con 0x7fa9d0105480 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.731+0000 7fa9d8600700 1 -- 172.21.15.63:0/2550723464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d0116160 con 0x7fa9d0105480 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.731+0000 7fa9ce000700 1 -- 172.21.15.63:0/2550723464 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9bc0037c0 con 0x7fa9d0105480 2024-04-25T19:05:19.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.731+0000 7fa9ce000700 1 -- 172.21.15.63:0/2550723464 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc004670 con 0x7fa9d0105480 2024-04-25T19:05:19.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.732+0000 7fa9d8600700 1 -- 172.21.15.63:0/2550723464 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d01a3ab0 con 0x7fa9d0105480 2024-04-25T19:05:19.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.737+0000 7fa9ce000700 1 -- 172.21.15.63:0/2550723464 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa9bc0041c0 con 0x7fa9d0105480 2024-04-25T19:05:19.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.739+0000 7fa9e2fe9b80 1 Processor -- start 2024-04-25T19:05:19.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.739+0000 7fa9e2fe9b80 1 -- start start 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9ce000700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa9b80a3e70 0x7fa9b80a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae100a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48858/0 (socket says 172.21.15.63:48858) 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9d6600700 1 -- 172.21.15.63:0/1999906715 learned_addr learned my addr 172.21.15.63:0/1999906715 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.740+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ffe8f90 con 0x55ae10096240 2024-04-25T19:05:19.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ff8f830 con 0x55ae100a10f0 2024-04-25T19:05:19.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ffe4f50 con 0x55ae10095920 2024-04-25T19:05:19.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae100a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 -- 172.21.15.63:0/1999906715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 msgr2=0x55ae100a34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae100a34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 -- 172.21.15.63:0/1999906715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 msgr2=0x55ae100a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d6600700 1 -- 172.21.15.63:0/1999906715 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0ff85780 con 0x55ae10095920 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.741+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.742+0000 7fa9d6600700 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800eb60 tx=0x7fa9c8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e494fa584f4801f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.743+0000 7fa9c2a00700 1 -- 172.21.15.63:0/1999906715 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c801f070 con 0x55ae10095920 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.743+0000 7fa9c2a00700 1 -- 172.21.15.63:0/1999906715 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c8009310 con 0x55ae10095920 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.743+0000 7fa9c2a00700 1 -- 172.21.15.63:0/1999906715 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8004290 con 0x55ae10095920 2024-04-25T19:05:19.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.743+0000 7fa9d5200700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa9b80a3e70 0x7fa9b80a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.743+0000 7fa9ce000700 1 -- 172.21.15.63:0/2550723464 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa9bc0e1ff0 con 0x7fa9d0105480 2024-04-25T19:05:19.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 msgr2=0x55ae10095d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800eb60 tx=0x7fa9c8009920 comp rx=0 tx=0).stop 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 shutdown_connections 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae10095d00 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae100a34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/1999906715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.744+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 >> 172.21.15.63:0/1999906715 conn(0x55ae0ffc46a0 msgr2=0x55ae10091dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:19.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.745+0000 7fa9d5200700 1 --2- 172.21.15.63:0/2550723464 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa9b80a3e70 0x7fa9b80a6330 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc0df7b0 tx=0x7fa9bc01a320 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.746+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 shutdown_connections 2024-04-25T19:05:19.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.746+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/1999906715 wait complete. 2024-04-25T19:05:19.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.746+0000 7fa9e2fe9b80 1 -- start start 2024-04-25T19:05:19.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 Processor -- start 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 0x55ae1009db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae100a10f0 0x55ae1009e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae100a3c10 con 0x55ae10095920 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ffa2960 con 0x55ae10096240 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ae100a3d90 con 0x55ae100a10f0 2024-04-25T19:05:19.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.747+0000 7fa9e2fe9b80 10 client.0 _ll_register_callbacks cb 0x55ae0ff81820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59354/0 (socket says 172.21.15.63:59354) 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 -- 172.21.15.63:0/2520834689 learned_addr learned my addr 172.21.15.63:0/2520834689 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 0x55ae1009db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae100a10f0 0x55ae1009e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 -- 172.21.15.63:0/2520834689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae100a10f0 msgr2=0x55ae1009e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae100a10f0 0x55ae1009e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 -- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 msgr2=0x55ae1009db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 0x55ae1009db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.748+0000 7fa9d6600700 1 -- 172.21.15.63:0/2520834689 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0ff369d0 con 0x55ae10095920 2024-04-25T19:05:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.749+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 0x55ae1009db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:05:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.749+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8011c90 tx=0x7fa9c800e1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b217efb5be42c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.749+0000 7fa9b7400700 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:05:19.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.750+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c801f040 con 0x55ae10095920 2024-04-25T19:05:19.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.750+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa9c8003a00 con 0x55ae10095920 2024-04-25T19:05:19.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.750+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9c8009310 con 0x55ae10095920 2024-04-25T19:05:19.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.751+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa9c802e050 con 0x55ae10095920 2024-04-25T19:05:19.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.752+0000 7fa9e2fe9b80 10 client.17136 fetch_fsmap learned FSMap version 59 2024-04-25T19:05:19.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.752+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ae0ff7fbf0 con 0x55ae10095920 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.753+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c8009310 con 0x55ae10095920 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.753+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 59) ==== 8342+0+0 (secure 0 0 0) 0x7fa9c8025070 con 0x55ae10095920 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 10 client.17136 fetch_fsmap finished waiting for FSMap version 59 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 10 client.17136 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 20 client.17136 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae100b0800 0x55ae100b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 4 client.17136 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:19.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae0fff6780 con 0x55ae100b0800 2024-04-25T19:05:19.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.754+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae100b0800 0x55ae100b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.755+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae100b0800 0x55ae100b2bf0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe99daf2d3877a89 server_cookie=f3dca41ee5c71a3d in_seq=0 out_seq=0 2024-04-25T19:05:19.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.755+0000 7fa9b7400700 10 client.17136 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:19.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.756+0000 7fa9b7400700 1 -- 172.21.15.63:0/2520834689 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae0fff6780 con 0x55ae100b0800 2024-04-25T19:05:19.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.756+0000 7fa9b7400700 10 client.17136 handle_command_reply: tid=0 2024-04-25T19:05:19.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.756+0000 7fa9e2fe9b80 1 client.17136 shutdown 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 msgr2=0x55ae100a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8011c90 tx=0x7fa9c800e1a0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 shutdown_connections 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae100b0800 0x55ae100b2bf0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae100a10f0 0x55ae1009e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10095920 0x55ae100a0880 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2520834689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10096240 0x55ae1009db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 >> 172.21.15.63:0/2520834689 conn(0x55ae0ffc46a0 msgr2=0x55ae100919c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9b7400700 0 client.17136 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:19.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.758+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 shutdown_connections 2024-04-25T19:05:19.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.759+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2520834689 wait complete. 2024-04-25T19:05:19.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.759+0000 7fa9e2fe9b80 20 client.17136 trim_cache size 0 max 16384 2024-04-25T19:05:19.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 Processor -- start 2024-04-25T19:05:19.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 -- start start 2024-04-25T19:05:19.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae100b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100b8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01adaa0 con 0x55ae10096240 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.791+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01ab300 con 0x55ae100a10f0 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9e2fe9b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01a9170 con 0x55ae10095920 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae100b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d5c00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43296/0 (socket says 172.21.15.63:43296) 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 -- 172.21.15.63:0/2872085020 learned_addr learned my addr 172.21.15.63:0/2872085020 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 -- 172.21.15.63:0/2872085020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 msgr2=0x55ae100b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae100b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 -- 172.21.15.63:0/2872085020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 msgr2=0x55ae100b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.792+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9d6600700 1 -- 172.21.15.63:0/2872085020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1009e030 con 0x55ae100a10f0 2024-04-25T19:05:19.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:05:19.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9d6600700 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800eb60 tx=0x7fa9c8011a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38545c3073903499 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9c2000700 1 -- 172.21.15.63:0/2872085020 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c801f070 con 0x55ae100a10f0 2024-04-25T19:05:19.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9c2000700 1 -- 172.21.15.63:0/2872085020 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c80092e0 con 0x55ae100a10f0 2024-04-25T19:05:19.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.793+0000 7fa9c2000700 1 -- 172.21.15.63:0/2872085020 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8007710 con 0x55ae100a10f0 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.794+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2872085020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 msgr2=0x55ae1009ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.794+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800eb60 tx=0x7fa9c8011a10 comp rx=0 tx=0).stop 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2872085020 shutdown_connections 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10095920 0x55ae100b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae100a10f0 0x55ae1009ae20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/2872085020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae10096240 0x55ae100b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2872085020 >> 172.21.15.63:0/2872085020 conn(0x55ae0ffc46a0 msgr2=0x55ae1009fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2872085020 shutdown_connections 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/2872085020 wait complete. 2024-04-25T19:05:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.795+0000 7fa9e2fe9b80 1 -- start start 2024-04-25T19:05:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.796+0000 7fa9e2fe9b80 1 Processor -- start 2024-04-25T19:05:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9e2fe9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9d6600700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43304/0 (socket says 172.21.15.63:43304) 2024-04-25T19:05:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9d6600700 1 -- 172.21.15.63:0/4271139086 learned_addr learned my addr 172.21.15.63:0/4271139086 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 0x55ae100a7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.797+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae100a10f0 0x55ae100a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.798+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01a97c0 con 0x55ae100a10f0 2024-04-25T19:05:19.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.798+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 0x55ae100a7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.798+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01aac30 con 0x55ae10095920 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.798+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d01a9e90 con 0x55ae10096240 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.798+0000 7fa9e2fe9b80 10 client.0 _ll_register_callbacks cb 0x55ae0ff81820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d6600700 1 -- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 msgr2=0x55ae100a7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d6600700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 0x55ae100a7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d6600700 1 -- 172.21.15.63:0/4271139086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae100a10f0 msgr2=0x55ae100a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d6600700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae100a10f0 0x55ae100a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d6600700 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae100a8440 con 0x55ae10095920 2024-04-25T19:05:19.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.799+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 0x55ae100a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9d6600700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009a00 tx=0x7fa9c8011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e8a60f363bb239 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9b7400700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c801f050 con 0x55ae10095920 2024-04-25T19:05:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa9c8003a00 con 0x55ae10095920 2024-04-25T19:05:19.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9c80092e0 con 0x55ae10095920 2024-04-25T19:05:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fa9c8042050 con 0x55ae10095920 2024-04-25T19:05:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9e2fe9b80 10 client.27380 fetch_fsmap learned FSMap version 59 2024-04-25T19:05:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.800+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ae10117a10 con 0x55ae10095920 2024-04-25T19:05:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.801+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9c80092e0 con 0x55ae10095920 2024-04-25T19:05:19.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8342+0+0 (secure 0 0 0) 0x7fa9c8025070 con 0x55ae10095920 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 10 client.27380 fetch_fsmap finished waiting for FSMap version 59 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 10 client.27380 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 20 client.27380 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae10117fa0 0x55ae1011a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 4 client.27380 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "e43e1e63", "await_for": 42.0}] 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "set_id": "e43e1e63", "await_for": 42.0}) -- 0x55ae0fff0820 con 0x55ae10117fa0 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.802+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae10117fa0 0x55ae1011a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.803+0000 7fa9d5c00700 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae10117fa0 0x55ae1011a3c0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d7b714e90ab4244 server_cookie=f2aa92c66e6dcbee in_seq=0 out_seq=0 2024-04-25T19:05:19.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:19.803+0000 7fa9b7400700 10 client.27380 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:20.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 113016 16.69 MB/sec execute 198 sec latency 595.353 ms 2024-04-25T19:05:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 -- 172.21.15.63:0/280688156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 msgr2=0x7f886010bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 --2- 172.21.15.63:0/280688156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f886010bd10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f884c009860 tx=0x7f884c02a3d0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 -- 172.21.15.63:0/280688156 shutdown_connections 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 --2- 172.21.15.63:0/280688156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f886010e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 --2- 172.21.15.63:0/280688156 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f886010bd10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 --2- 172.21.15.63:0/280688156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f88601032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 -- 172.21.15.63:0/280688156 >> 172.21.15.63:0/280688156 conn(0x7f88600fc770 msgr2=0x7f88600feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 -- 172.21.15.63:0/280688156 shutdown_connections 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.426+0000 7f8864c00640 1 -- 172.21.15.63:0/280688156 wait complete. 2024-04-25T19:05:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.427+0000 7f8864c00640 1 Processor -- start 2024-04-25T19:05:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 -- start start 2024-04-25T19:05:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f8860100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f88601010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8860112010 con 0x7f8860102f10 2024-04-25T19:05:20.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8860111e90 con 0x7f886010c330 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f8864c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8860112190 con 0x7f8860103830 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885ea00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f88601010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f8860100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885fe00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885f400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f8860100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59396/0 (socket says 172.21.15.63:59396) 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885f400640 1 -- 172.21.15.63:0/3710509359 learned_addr learned my addr 172.21.15.63:0/3710509359 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43324/0 (socket says 172.21.15.63:43324) 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.428+0000 7f885ea00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f88601010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48882/0 (socket says 172.21.15.63:48882) 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 -- 172.21.15.63:0/3710509359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 msgr2=0x7f88601010a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f88601010a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 -- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 msgr2=0x7f8860100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f8860100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 -- 172.21.15.63:0/3710509359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88601a6ab0 con 0x7f886010c330 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.429+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f885800ef00 tx=0x7f885800c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.430+0000 7f8857e00640 1 -- 172.21.15.63:0/3710509359 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858004b20 con 0x7f886010c330 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.430+0000 7f8857e00640 1 -- 172.21.15.63:0/3710509359 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8858003e20 con 0x7f886010c330 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.430+0000 7f8857e00640 1 -- 172.21.15.63:0/3710509359 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800b570 con 0x7f886010c330 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.430+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88601a6da0 con 0x7f886010c330 2024-04-25T19:05:20.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.430+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601a7270 con 0x7f886010c330 2024-04-25T19:05:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.435+0000 7f8868ffa740 1 Processor -- start 2024-04-25T19:05:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.436+0000 7f8857e00640 1 -- 172.21.15.63:0/3710509359 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f885801d020 con 0x7f886010c330 2024-04-25T19:05:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.436+0000 7f8868ffa740 1 -- start start 2024-04-25T19:05:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.436+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 0x563502656d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.436+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 0x563502661650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x563502663f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f8868ffa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5635025bc900 con 0x563502661b90 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f8868ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563502562ef0 con 0x5635026569b0 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f8868ffa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x5635025b88c0 con 0x5635026572d0 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f885ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 0x563502656d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f885ea00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 0x563502656d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43334/0 (socket says 172.21.15.63:43334) 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f885ea00640 1 -- 172.21.15.63:0/858837767 learned_addr learned my addr 172.21.15.63:0/858837767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.437+0000 7f885ea00640 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x563502663f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f885f400640 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 0x563502661650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f8857e00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 0x7f88440a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f8857e00640 1 -- 172.21.15.63:0/3710509359 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f88580d04e0 con 0x7f886010c330 2024-04-25T19:05:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f885f400640 1 -- 172.21.15.63:0/858837767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 msgr2=0x563502656d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f885f400640 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 0x563502656d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.438+0000 7f885f400640 1 -- 172.21.15.63:0/858837767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 msgr2=0x563502663f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.441+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 0x7f88440a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.441+0000 7f885f400640 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x563502663f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.443 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.441+0000 7f885f400640 1 -- 172.21.15.63:0/858837767 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635022cf210 con 0x5635026572d0 2024-04-25T19:05:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.443+0000 7f885fe00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 0x7f88440a6250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88580cdca0 tx=0x7f8858009b70 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.443+0000 7f885f400640 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 0x563502661650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f88500058c0 tx=0x7f88500107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db9e1c73e447074b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.443+0000 7f884be00640 1 -- 172.21.15.63:0/858837767 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850004230 con 0x5635026572d0 2024-04-25T19:05:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.443+0000 7f884be00640 1 -- 172.21.15.63:0/858837767 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885000be90 con 0x5635026572d0 2024-04-25T19:05:20.444 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.443+0000 7f884be00640 1 -- 172.21.15.63:0/858837767 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850011680 con 0x5635026572d0 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.444+0000 7f8868ffa740 1 -- 172.21.15.63:0/858837767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 msgr2=0x563502661650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.444+0000 7f8868ffa740 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 0x563502661650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f88500058c0 tx=0x7f88500107a0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 -- 172.21.15.63:0/858837767 shutdown_connections 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x563502663f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026572d0 0x563502661650 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 --2- 172.21.15.63:0/858837767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026569b0 0x563502656d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.446 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 -- 172.21.15.63:0/858837767 >> 172.21.15.63:0/858837767 conn(0x563502652ec0 msgr2=0x5635026532a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 -- 172.21.15.63:0/858837767 shutdown_connections 2024-04-25T19:05:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.445+0000 7f8868ffa740 1 -- 172.21.15.63:0/858837767 wait complete. 2024-04-25T19:05:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.446+0000 7f8868ffa740 1 -- start start 2024-04-25T19:05:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.446+0000 7f8868ffa740 1 Processor -- start 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026569b0 0x56350265ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x56350265f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5635026647c0 con 0x563502661b90 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563502576310 con 0x5635026572d0 2024-04-25T19:05:20.451 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x563502664940 con 0x5635026569b0 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.448+0000 7f8868ffa740 10 client.0 _ll_register_callbacks cb 0x563502544cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:43346/0 (socket says 172.21.15.63:43346) 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 -- 172.21.15.63:0/1818596488 learned_addr learned my addr 172.21.15.63:0/1818596488 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 -- 172.21.15.63:0/1818596488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026569b0 msgr2=0x56350265ea90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026569b0 0x56350265ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.450+0000 7f885f400640 1 -- 172.21.15.63:0/1818596488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 msgr2=0x56350265f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.451+0000 7f885ea00640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x56350265f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.453 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.452+0000 7f885f400640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x56350265f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.452+0000 7f885f400640 1 -- 172.21.15.63:0/1818596488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56350239b620 con 0x5635026572d0 2024-04-25T19:05:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.452+0000 7f885ea00640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x56350265f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:05:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.453+0000 7f885f400640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8850003dd0 tx=0x7f8850004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eb1a18a2503a735 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.453+0000 7f8848c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.456+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885002b060 con 0x5635026572d0 2024-04-25T19:05:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.456+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f88500035f0 con 0x5635026572d0 2024-04-25T19:05:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.456+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f885000be90 con 0x5635026572d0 2024-04-25T19:05:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.457+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f88500037f0 con 0x5635026572d0 2024-04-25T19:05:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.457+0000 7f8868ffa740 10 client.27392 fetch_fsmap learned FSMap version 59 2024-04-25T19:05:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.457+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x5635026791d0 con 0x5635026572d0 2024-04-25T19:05:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f885000be90 con 0x5635026572d0 2024-04-25T19:05:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8342+0+0 (secure 0 0 0) 0x7f8850030420 con 0x5635026572d0 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8868ffa740 10 client.27392 fetch_fsmap finished waiting for FSMap version 59 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8868ffa740 10 client.27392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8868ffa740 20 client.27392 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56350267a580 0x56350267c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.459+0000 7f885ea00640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56350267a580 0x56350267c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.458+0000 7f8868ffa740 4 client.27392 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.459+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635025d08b0 con 0x56350267a580 2024-04-25T19:05:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.462+0000 7f885ea00640 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56350267a580 0x56350267c970 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b897009120f36918 server_cookie=88e8e16abead0043 in_seq=0 out_seq=0 2024-04-25T19:05:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.463+0000 7f8848c00640 10 client.27392 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.468+0000 7f8848c00640 1 -- 172.21.15.63:0/1818596488 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635025d08b0 con 0x56350267a580 2024-04-25T19:05:20.469 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.468+0000 7f8848c00640 10 client.27392 handle_command_reply: tid=0 2024-04-25T19:05:20.470 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.468+0000 7f8868ffa740 1 client.27392 shutdown 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 msgr2=0x56350265efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8850003dd0 tx=0x7f8850004230 comp rx=0 tx=0).stop 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 shutdown_connections 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x56350267a580 0x56350267c970 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026572d0 0x56350265efd0 secure :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8850003dd0 tx=0x7f8850004230 comp rx=0 tx=0).stop 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563502661b90 0x56350265f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 --2- 172.21.15.63:0/1818596488 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026569b0 0x56350265ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 >> 172.21.15.63:0/1818596488 conn(0x563502652ec0 msgr2=0x563502658bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8848c00640 0 client.27392 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:20.471 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.469+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 shutdown_connections 2024-04-25T19:05:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.470+0000 7f8868ffa740 1 -- 172.21.15.63:0/1818596488 wait complete. 2024-04-25T19:05:20.472 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.470+0000 7f8868ffa740 20 client.27392 trim_cache size 0 max 16384 2024-04-25T19:05:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.501+0000 7f8868ffa740 1 Processor -- start 2024-04-25T19:05:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.501+0000 7f8868ffa740 1 -- start start 2024-04-25T19:05:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x56350266d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026612d0 0x5635026824b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88601ae410 con 0x5635026569b0 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88601abcb0 con 0x5635026612d0 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.502+0000 7f8868ffa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88601a9b20 con 0x563502661b90 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:48910/0 (socket says 172.21.15.63:48910) 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 -- 172.21.15.63:0/3050482981 learned_addr learned my addr 172.21.15.63:0/3050482981 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885ea00640 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x56350266d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 -- 172.21.15.63:0/3050482981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026612d0 msgr2=0x5635026824b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026612d0 0x5635026824b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 -- 172.21.15.63:0/3050482981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 msgr2=0x56350266d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x56350266d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.505 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885f400640 1 -- 172.21.15.63:0/3050482981 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56350265a4e0 con 0x563502661b90 2024-04-25T19:05:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.503+0000 7f885ea00640 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x56350266d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:05:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.504+0000 7f885f400640 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8850030960 tx=0x7f8850005640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92945d5d4aac376d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.505+0000 7f884b400640 1 -- 172.21.15.63:0/3050482981 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885002b050 con 0x563502661b90 2024-04-25T19:05:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.505+0000 7f884b400640 1 -- 172.21.15.63:0/3050482981 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88500035f0 con 0x563502661b90 2024-04-25T19:05:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.505+0000 7f884b400640 1 -- 172.21.15.63:0/3050482981 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850035080 con 0x563502661b90 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 -- 172.21.15.63:0/3050482981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 msgr2=0x563502681f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8850030960 tx=0x7f8850005640 comp rx=0 tx=0).stop 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 -- 172.21.15.63:0/3050482981 shutdown_connections 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026612d0 0x5635026824b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x563502661b90 0x563502681f70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 --2- 172.21.15.63:0/3050482981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x56350266d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.508 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.506+0000 7f8868ffa740 1 -- 172.21.15.63:0/3050482981 >> 172.21.15.63:0/3050482981 conn(0x563502652ec0 msgr2=0x563502659dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:20.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.507+0000 7f8868ffa740 1 -- 172.21.15.63:0/3050482981 shutdown_connections 2024-04-25T19:05:20.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.507+0000 7f8868ffa740 1 -- 172.21.15.63:0/3050482981 wait complete. 2024-04-25T19:05:20.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.507+0000 7f8868ffa740 1 -- start start 2024-04-25T19:05:20.509 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.507+0000 7f8868ffa740 1 Processor -- start 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.508+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x563502667db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026612d0 0x5635026682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 0x5635026d9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f88601aa170 con 0x5635026569b0 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88601ab5e0 con 0x5635026d9830 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f88601aa840 con 0x5635026612d0 2024-04-25T19:05:20.510 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f8868ffa740 10 client.0 _ll_register_callbacks cb 0x7f88500056d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.509+0000 7f885ea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x563502667db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.510+0000 7f885ea00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x563502667db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59430/0 (socket says 172.21.15.63:59430) 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.510+0000 7f885ea00640 1 -- 172.21.15.63:0/901370126 learned_addr learned my addr 172.21.15.63:0/901370126 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.510+0000 7f885ea00640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 0x5635026d9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.510+0000 7f885f400640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026612d0 0x5635026682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.511+0000 7f885ea00640 1 -- 172.21.15.63:0/901370126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026612d0 msgr2=0x5635026682f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.512 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.511+0000 7f885ea00640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026612d0 0x5635026682f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.511+0000 7f885ea00640 1 -- 172.21.15.63:0/901370126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 msgr2=0x563502667db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:20.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.511+0000 7f885ea00640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x563502667db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.511+0000 7f885ea00640 1 -- 172.21.15.63:0/901370126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635026da270 con 0x5635026d9830 2024-04-25T19:05:20.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.512+0000 7f885ea00640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 0x5635026d9cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f884c02a8f0 tx=0x7f884c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9263ae7dd31d4c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:20.513 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.512+0000 7f8848c00640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:20.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.515+0000 7fa9b7400700 1 -- 172.21.15.63:0/4271139086 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: -110 ) ==== 8+0+592 (crc 0 0 0) 0x55ae0fff0820 con 0x55ae10117fa0 2024-04-25T19:05:20.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.515+0000 7fa9b7400700 10 client.27380 handle_command_reply: tid=0 2024-04-25T19:05:20.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.515+0000 7fa9e2fe9b80 1 client.27380 shutdown 2024-04-25T19:05:20.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.516+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c030580 con 0x5635026d9830 2024-04-25T19:05:20.517 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.516+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7f884c030720 con 0x5635026d9830 2024-04-25T19:05:20.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 msgr2=0x55ae100a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:20.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009a00 tx=0x7fa9c8011930 comp rx=0 tx=0).stop 2024-04-25T19:05:20.518 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.516+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88500035f0 con 0x5635026d9830 2024-04-25T19:05:20.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 shutdown_connections 2024-04-25T19:05:20.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ae10117fa0 0x55ae1011a3c0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae100a10f0 0x55ae100a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ae10095920 0x55ae100a9360 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9b7400700 0 client.27380 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 --2- 172.21.15.63:0/4271139086 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ae10096240 0x55ae100a7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 >> 172.21.15.63:0/4271139086 conn(0x55ae0ffc46a0 msgr2=0x55ae1009f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.516+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 shutdown_connections 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.517+0000 7fa9e2fe9b80 1 -- 172.21.15.63:0/4271139086 wait complete. 2024-04-25T19:05:20.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:05:20.517+0000 7fa9e2fe9b80 20 client.27380 trim_cache size 0 max 16384 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 59, 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 50, 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": { 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "e43e1e63": { 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "version": 50, 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "age_ref": 6.2, 2024-04-25T19:05:20.520 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "TIMEDOUT", 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 0.0 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "timeout": 60.0, 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "expiration": 74.0, 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "members": { 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "file:/": { 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "excluded": false, 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "state": { 2024-04-25T19:05:20.521 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "QUIESCING", 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: "age": 60.0 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:05:20.522 INFO:teuthology.orchestra.run.smithi063.stderr:Error ETIMEDOUT: 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.520+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f884c030980 con 0x5635026d9830 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.520+0000 7f8868ffa740 10 client.27395 fetch_fsmap learned FSMap version 59 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.520+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x5635026e1590 con 0x5635026d9830 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.521+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88500035f0 con 0x5635026d9830 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.521+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 59) ==== 8342+0+0 (secure 0 0 0) 0x7f884c005040 con 0x5635026d9830 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.521+0000 7f8868ffa740 10 client.27395 fetch_fsmap finished waiting for FSMap version 59 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.521+0000 7f8868ffa740 10 client.27395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:20.523 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.521+0000 7f8868ffa740 20 client.27395 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5635026e1d40 0x5635026e4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f8868ffa740 4 client.27395 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56350267cd60 con 0x5635026e1d40 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f885f400640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5635026e1d40 0x5635026e4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f885f400640 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5635026e1d40 0x5635026e4160 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7cbdbad125a6470 server_cookie=f467a48d2f2dc63d in_seq=0 out_seq=0 2024-04-25T19:05:20.524 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:20.522+0000 7f8848c00640 10 client.27395 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:20 smithi114 ceph-mon[32988]: pgmap v594: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-04-25T19:05:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:20 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:20.873 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:20 smithi063 ceph-mon[28006]: pgmap v594: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-04-25T19:05:20.873 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:20 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:20.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:20 smithi070 ceph-mon[32820]: pgmap v594: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 219 op/s 2024-04-25T19:05:20.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:20 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:21.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 113425 16.67 MB/sec execute 199 sec latency 1595.411 ms 2024-04-25T19:05:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-25T19:05:21.303 ERROR:tasks.quiescer.fs.[cephfs]:Couldn't quiesce root with rc: 110 (ETIMEDOUT), stdout: { "epoch": 59, "leader": 24533, "set_version": 50, "sets": { "e43e1e63": { "version": 50, "age_ref": 6.2, "state": { "name": "TIMEDOUT", "age": 0.0 }, "timeout": 60.0, "expiration": 74.0, "members": { "file:/": { "excluded": false, "state": { "name": "QUIESCING", "age": 60.0 } } } } } } 2024-04-25T19:05:21.303 ERROR:tasks.quiescer.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/quiescer.py", line 346, in _run self.do_quiesce(d) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/quiescer.py", line 301, in do_quiesce raise RuntimeError(f"Error quiescing set '{set_id}': {rcinfo}") RuntimeError: Error quiescing set 'e43e1e63': 110 (ETIMEDOUT) 2024-04-25T19:05:22.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 113963 16.75 MB/sec execute 200 sec latency 2595.468 ms 2024-04-25T19:05:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:22 smithi114 ceph-mon[32988]: pgmap v595: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:22.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:22 smithi063 ceph-mon[28006]: pgmap v595: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:22.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:22 smithi070 ceph-mon[32820]: pgmap v595: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:23.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 114458 16.78 MB/sec execute 201 sec latency 3595.535 ms 2024-04-25T19:05:24.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 114999 16.85 MB/sec execute 202 sec latency 4595.586 ms 2024-04-25T19:05:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:24 smithi114 ceph-mon[32988]: pgmap v596: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:24.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:24 smithi063 ceph-mon[28006]: pgmap v596: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:24.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:24 smithi070 ceph-mon[32820]: pgmap v596: 97 pgs: 97 active+clean; 952 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.5 MiB/s wr, 212 op/s 2024-04-25T19:05:25.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 115595 16.95 MB/sec execute 203 sec latency 5595.634 ms 2024-04-25T19:05:26.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 116066 16.97 MB/sec execute 204 sec latency 6595.688 ms 2024-04-25T19:05:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:26 smithi114 ceph-mon[32988]: pgmap v597: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 426 op/s 2024-04-25T19:05:26.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:26 smithi063 ceph-mon[28006]: pgmap v597: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 426 op/s 2024-04-25T19:05:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:26 smithi070 ceph-mon[32820]: pgmap v597: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 426 op/s 2024-04-25T19:05:27.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 116601 17.01 MB/sec execute 205 sec latency 7595.769 ms 2024-04-25T19:05:28.113 INFO:tasks.workunit.client.0.smithi063.stdout: 10 117176 17.11 MB/sec execute 206 sec latency 8595.839 ms 2024-04-25T19:05:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:28 smithi114 ceph-mon[32988]: pgmap v598: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 422 op/s 2024-04-25T19:05:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:28 smithi063 ceph-mon[28006]: pgmap v598: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 422 op/s 2024-04-25T19:05:28.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:28 smithi070 ceph-mon[32820]: pgmap v598: 97 pgs: 97 active+clean; 979 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5.1 MiB/s wr, 422 op/s 2024-04-25T19:05:29.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 117672 17.13 MB/sec execute 207 sec latency 9596.457 ms 2024-04-25T19:05:30.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 118203 17.19 MB/sec execute 208 sec latency 10596.524 ms 2024-04-25T19:05:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:30 smithi114 ceph-mon[32988]: pgmap v599: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.9 MiB/s wr, 587 op/s 2024-04-25T19:05:30.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:30 smithi063 ceph-mon[28006]: pgmap v599: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.9 MiB/s wr, 587 op/s 2024-04-25T19:05:30.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:30 smithi070 ceph-mon[32820]: pgmap v599: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 7.9 MiB/s wr, 587 op/s 2024-04-25T19:05:31.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 118679 17.25 MB/sec execute 209 sec latency 11596.607 ms 2024-04-25T19:05:32.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 119091 17.24 MB/sec execute 210 sec latency 12596.692 ms 2024-04-25T19:05:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:32 smithi114 ceph-mon[32988]: pgmap v600: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:32.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:32 smithi063 ceph-mon[28006]: pgmap v600: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:32.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:32 smithi070 ceph-mon[32820]: pgmap v600: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:33.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 119701 17.34 MB/sec execute 211 sec latency 13596.746 ms 2024-04-25T19:05:34.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 120310 17.39 MB/sec execute 212 sec latency 14596.797 ms 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.201+0000 7f8848c00640 1 -- 172.21.15.63:0/901370126 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56350267cd60 con 0x5635026e1d40 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.201+0000 7f8848c00640 10 client.27395 handle_command_reply: tid=0 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.201+0000 7f8868ffa740 1 client.27395 shutdown 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 msgr2=0x5635026d9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 0x5635026d9cb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f884c02a8f0 tx=0x7f884c005c90 comp rx=0 tx=0).stop 2024-04-25T19:05:34.203 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 shutdown_connections 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x5635026e1d40 0x5635026e4160 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5635026d9830 0x5635026d9cb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x5635026612d0 0x5635026682f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 --2- 172.21.15.63:0/901370126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5635026569b0 0x563502667db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 >> 172.21.15.63:0/901370126 conn(0x563502652ec0 msgr2=0x563502658360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.202+0000 7f8848c00640 0 client.27395 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:34.204 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.203+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 shutdown_connections 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.203+0000 7f8868ffa740 1 -- 172.21.15.63:0/901370126 wait complete. 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.203+0000 7f8868ffa740 20 client.27395 trim_cache size 0 max 16384 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:05:34.205 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 msgr2=0x7f88440a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 0x7f88440a6250 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f88580cdca0 tx=0x7f8858009b70 comp rx=0 tx=0).stop 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 msgr2=0x7f88601015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f885800ef00 tx=0x7f885800c740 comp rx=0 tx=0).stop 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 shutdown_connections 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f88440a3d90 0x7f88440a6250 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.206 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f886010c330 0x7f88601015e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8860103830 0x7f88601010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 --2- 172.21.15.63:0/3710509359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8860102f10 0x7f8860100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.204+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 >> 172.21.15.63:0/3710509359 conn(0x7f88600fc770 msgr2=0x7f88600fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.205+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 shutdown_connections 2024-04-25T19:05:34.207 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:34.205+0000 7f8864c00640 1 -- 172.21.15.63:0/3710509359 wait complete. 2024-04-25T19:05:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:34 smithi114 ceph-mon[32988]: pgmap v601: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:34.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:34 smithi063 ceph-mon[28006]: pgmap v601: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:34.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:34 smithi070 ceph-mon[32820]: pgmap v601: 97 pgs: 97 active+clean; 1010 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 378 op/s 2024-04-25T19:05:35.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 120792 17.45 MB/sec execute 213 sec latency 14697.826 ms 2024-04-25T19:05:35.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:35 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:35.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:35 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:36.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:35 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:36.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 121389 17.54 MB/sec execute 214 sec latency 57.056 ms 2024-04-25T19:05:36.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:36 smithi063 ceph-mon[28006]: pgmap v602: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 559 op/s 2024-04-25T19:05:36.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:36 smithi070 ceph-mon[32820]: pgmap v602: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 559 op/s 2024-04-25T19:05:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:36 smithi114 ceph-mon[32988]: pgmap v602: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 8.3 MiB/s wr, 559 op/s 2024-04-25T19:05:37.114 INFO:tasks.workunit.client.0.smithi063.stdout: 10 121852 17.57 MB/sec execute 215 sec latency 103.682 ms 2024-04-25T19:05:38.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 122232 17.57 MB/sec execute 216 sec latency 193.259 ms 2024-04-25T19:05:38.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:38 smithi063 ceph-mon[28006]: pgmap v603: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 345 op/s 2024-04-25T19:05:38.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:38 smithi070 ceph-mon[32820]: pgmap v603: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 345 op/s 2024-04-25T19:05:39.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:38 smithi114 ceph-mon[32988]: pgmap v603: 97 pgs: 97 active+clean; 1.0 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 345 op/s 2024-04-25T19:05:39.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 122622 17.56 MB/sec execute 217 sec latency 240.085 ms 2024-04-25T19:05:40.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 123187 17.61 MB/sec execute 218 sec latency 110.098 ms 2024-04-25T19:05:40.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:40 smithi063 ceph-mon[28006]: pgmap v604: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 499 op/s 2024-04-25T19:05:40.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:40 smithi070 ceph-mon[32820]: pgmap v604: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 499 op/s 2024-04-25T19:05:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:40 smithi114 ceph-mon[32988]: pgmap v604: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 8.3 MiB/s wr, 499 op/s 2024-04-25T19:05:41.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 123801 17.68 MB/sec execute 219 sec latency 80.401 ms 2024-04-25T19:05:42.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 124396 17.77 MB/sec execute 220 sec latency 89.788 ms 2024-04-25T19:05:42.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:42 smithi114 ceph-mon[32988]: pgmap v605: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:42.879 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:42 smithi063 ceph-mon[28006]: pgmap v605: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:42.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:42 smithi070 ceph-mon[32820]: pgmap v605: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:43.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 124922 17.81 MB/sec execute 221 sec latency 79.256 ms 2024-04-25T19:05:44.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 125440 17.90 MB/sec execute 222 sec latency 74.147 ms 2024-04-25T19:05:44.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:44 smithi063 ceph-mon[28006]: pgmap v606: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:44.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:44 smithi070 ceph-mon[32820]: pgmap v606: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:45.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:44 smithi114 ceph-mon[32988]: pgmap v606: 97 pgs: 97 active+clean; 1.0 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 5.5 MiB/s wr, 335 op/s 2024-04-25T19:05:45.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 125785 17.88 MB/sec execute 223 sec latency 219.990 ms 2024-04-25T19:05:46.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 126264 17.89 MB/sec execute 224 sec latency 138.220 ms 2024-04-25T19:05:46.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:46 smithi063 ceph-mon[28006]: pgmap v607: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 491 op/s 2024-04-25T19:05:46.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:46 smithi070 ceph-mon[32820]: pgmap v607: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 491 op/s 2024-04-25T19:05:47.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:46 smithi114 ceph-mon[32988]: pgmap v607: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.9 MiB/s wr, 491 op/s 2024-04-25T19:05:47.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 126833 17.96 MB/sec execute 225 sec latency 139.559 ms 2024-04-25T19:05:47.374 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 -- 172.21.15.63:0/2425914816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c103b00 msgr2=0x7fa60c10bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 --2- 172.21.15.63:0/2425914816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c103b00 0x7fa60c10bfe0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009860 tx=0x7fa5f802a460 comp rx=0 tx=0).stop 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 -- 172.21.15.63:0/2425914816 shutdown_connections 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 --2- 172.21.15.63:0/2425914816 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c103b00 0x7fa60c10bfe0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 --2- 172.21.15.63:0/2425914816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 0x7fa60c10e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 --2- 172.21.15.63:0/2425914816 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c1031e0 0x7fa60c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 -- 172.21.15.63:0/2425914816 >> 172.21.15.63:0/2425914816 conn(0x7fa60c077fd0 msgr2=0x7fa60c0783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 -- 172.21.15.63:0/2425914816 shutdown_connections 2024-04-25T19:05:47.375 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.373+0000 7fa610c00640 1 -- 172.21.15.63:0/2425914816 wait complete. 2024-04-25T19:05:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.374+0000 7fa610c00640 1 Processor -- start 2024-04-25T19:05:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.374+0000 7fa610c00640 1 -- start start 2024-04-25T19:05:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 0x7fa60c1a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.376 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 0x7fa60c19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1121e0 con 0x7fa60c10c520 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c112060 con 0x7fa60c1031e0 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa610c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c112360 con 0x7fa60c103b00 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa609e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa609e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57990/0 (socket says 172.21.15.63:57990) 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa609400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 0x7fa60c1a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.377 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa609e00640 1 -- 172.21.15.63:0/3399273864 learned_addr learned my addr 172.21.15.63:0/3399273864 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.375+0000 7fa60a800640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 0x7fa60c19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 -- 172.21.15.63:0/3399273864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 msgr2=0x7fa60c1a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.378 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 -- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 msgr2=0x7fa60c19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 0x7fa60c19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 -- 172.21.15.63:0/3399273864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60c19afd0 con 0x7fa60c103b00 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609e00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa609400640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 0x7fa60c1a0b80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802a980 tx=0x7fa5f8030fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa602a00640 1 -- 172.21.15.63:0/3399273864 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8040070 con 0x7fa60c103b00 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa602a00640 1 -- 172.21.15.63:0/3399273864 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f8005570 con 0x7fa60c103b00 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60c19b260 con 0x7fa60c103b00 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.376+0000 7fa602a00640 1 -- 172.21.15.63:0/3399273864 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f80046c0 con 0x7fa60c103b00 2024-04-25T19:05:47.379 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.377+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60c1a7490 con 0x7fa60c103b00 2024-04-25T19:05:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.380+0000 7fa602a00640 1 -- 172.21.15.63:0/3399273864 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa5f8043070 con 0x7fa60c103b00 2024-04-25T19:05:47.381 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.380+0000 7fa6139be740 1 Processor -- start 2024-04-25T19:05:47.382 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.380+0000 7fa6139be740 1 -- start start 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb41e900 con 0x55ccbb4c3a00 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb3c4ef0 con 0x55ccbb4b8820 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa6139be740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb41a8c0 con 0x55ccbb4b9140 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa60a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa60a800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57992/0 (socket says 172.21.15.63:57992) 2024-04-25T19:05:47.383 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa60a800640 1 -- 172.21.15.63:0/3600816344 learned_addr learned my addr 172.21.15.63:0/3600816344 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa609e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.381+0000 7fa609400640 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 -- 172.21.15.63:0/3600816344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b9140 msgr2=0x55ccbb4c34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 -- 172.21.15.63:0/3600816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4c3a00 msgr2=0x55ccbb4c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.384 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 -- 172.21.15.63:0/3600816344 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccbb131210 con 0x55ccbb4b8820 2024-04-25T19:05:47.385 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa60a800640 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa604009f50 tx=0x7fa6040118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a09fcaa232ea526 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa602a00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 0x7fa5f00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa602a00640 1 -- 172.21.15.63:0/3399273864 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa5f80f1310 con 0x7fa60c103b00 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa609e00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 0x7fa5f00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa5f7000640 1 -- 172.21.15.63:0/3600816344 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604004d20 con 0x55ccbb4b8820 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa5f7000640 1 -- 172.21.15.63:0/3600816344 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa604004050 con 0x55ccbb4b8820 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.382+0000 7fa5f7000640 1 -- 172.21.15.63:0/3600816344 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604005280 con 0x55ccbb4b8820 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 -- 172.21.15.63:0/3600816344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 msgr2=0x55ccbb4b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa609e00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 0x7fa5f00a6300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009fd0 tx=0x7fa5fc009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa604009f50 tx=0x7fa6040118d0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 -- 172.21.15.63:0/3600816344 shutdown_connections 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.386 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.383+0000 7fa6139be740 1 --2- 172.21.15.63:0/3600816344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b8820 0x55ccbb4b8c00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.384+0000 7fa6139be740 1 -- 172.21.15.63:0/3600816344 >> 172.21.15.63:0/3600816344 conn(0x55ccbb4b4d30 msgr2=0x55ccbb4b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.384+0000 7fa6139be740 1 -- 172.21.15.63:0/3600816344 shutdown_connections 2024-04-25T19:05:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.384+0000 7fa6139be740 1 -- 172.21.15.63:0/3600816344 wait complete. 2024-04-25T19:05:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.384+0000 7fa6139be740 1 -- start start 2024-04-25T19:05:47.387 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.384+0000 7fa6139be740 1 Processor -- start 2024-04-25T19:05:47.389 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4c09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb4c6630 con 0x55ccbb4b8820 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb3d8310 con 0x55ccbb4b9140 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55ccbb4c67b0 con 0x55ccbb4c3a00 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa6139be740 10 client.0 _ll_register_callbacks cb 0x55ccbb3a6cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa609400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa60a800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa609400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:57994/0 (socket says 172.21.15.63:57994) 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa609400640 1 -- 172.21.15.63:0/1039434372 learned_addr learned my addr 172.21.15.63:0/1039434372 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.385+0000 7fa609e00640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4c09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 -- 172.21.15.63:0/1039434372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 msgr2=0x55ccbb4c1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.390 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 -- 172.21.15.63:0/1039434372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 msgr2=0x55ccbb4c09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4c09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 -- 172.21.15.63:0/1039434372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccbb1fd620 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa60a800640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa609400640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80b9720 tx=0x7fa5f8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c410fa1b5cf2bef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa5eb400640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f804a040 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa5f80c1150 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa604004050 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fa5f80bce00 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.386+0000 7fa6139be740 10 client.27407 fetch_fsmap learned FSMap version 61 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55ccbb4d28c0 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa604004050 con 0x55ccbb4b9140 2024-04-25T19:05:47.391 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7fa5f80bd520 con 0x55ccbb4b9140 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 10 client.27407 fetch_fsmap finished waiting for FSMap version 61 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 10 client.27407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 20 client.27407 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb4d3d20 0x55ccbb4d6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 4 client.27407 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.387+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccbb4328b0 con 0x55ccbb4d3d20 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.388+0000 7fa609e00640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb4d3d20 0x55ccbb4d6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.388+0000 7fa609e00640 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb4d3d20 0x55ccbb4d6190 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a58eb571e49251 server_cookie=419fa52429a87251 in_seq=0 out_seq=0 2024-04-25T19:05:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.388+0000 7fa5eb400640 10 client.27407 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa5eb400640 1 -- 172.21.15.63:0/1039434372 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ccbb4328b0 con 0x55ccbb4d3d20 2024-04-25T19:05:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa5eb400640 10 client.27407 handle_command_reply: tid=0 2024-04-25T19:05:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 client.27407 shutdown 2024-04-25T19:05:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 msgr2=0x55ccbb4c0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80b9720 tx=0x7fa5f8005f90 comp rx=0 tx=0).stop 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 shutdown_connections 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb4d3d20 0x55ccbb4d6190 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c0f00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 --2- 172.21.15.63:0/1039434372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4c09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 >> 172.21.15.63:0/1039434372 conn(0x55ccbb4b4d30 msgr2=0x55ccbb4baa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa5eb400640 0 client.27407 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.390+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 shutdown_connections 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.391+0000 7fa6139be740 1 -- 172.21.15.63:0/1039434372 wait complete. 2024-04-25T19:05:47.394 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.391+0000 7fa6139be740 20 client.27407 trim_cache size 0 max 16384 2024-04-25T19:05:47.421 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.420+0000 7fa6139be740 1 Processor -- start 2024-04-25T19:05:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.420+0000 7fa6139be740 1 -- start start 2024-04-25T19:05:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4dbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1b2780 con 0x55ccbb4b8820 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1affc0 con 0x55ccbb4b9140 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa6139be740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1ade30 con 0x55ccbb4c3a00 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa609400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa609400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:58004/0 (socket says 172.21.15.63:58004) 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa609400640 1 -- 172.21.15.63:0/572121368 learned_addr learned my addr 172.21.15.63:0/572121368 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa60a800640 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4dbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.421+0000 7fa609e00640 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 -- 172.21.15.63:0/572121368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 msgr2=0x55ccbb4dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4dbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 -- 172.21.15.63:0/572121368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 msgr2=0x55ccbb4cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 -- 172.21.15.63:0/572121368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccbb4bf970 con 0x55ccbb4b9140 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa609400640 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80c5f00 tx=0x7fa5f8032fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39287acbbc8aa0e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa5f6600640 1 -- 172.21.15.63:0/572121368 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f804a030 con 0x55ccbb4b9140 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa5f6600640 1 -- 172.21.15.63:0/572121368 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f80d3e50 con 0x55ccbb4b9140 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.422+0000 7fa5f6600640 1 -- 172.21.15.63:0/572121368 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f80bcc20 con 0x55ccbb4b9140 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 -- 172.21.15.63:0/572121368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 msgr2=0x55ccbb4b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80c5f00 tx=0x7fa5f8032fb0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 -- 172.21.15.63:0/572121368 shutdown_connections 2024-04-25T19:05:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4b9140 0x55ccbb4b3e90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4dbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.423+0000 7fa6139be740 1 --2- 172.21.15.63:0/572121368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.424+0000 7fa6139be740 1 -- 172.21.15.63:0/572121368 >> 172.21.15.63:0/572121368 conn(0x55ccbb4b4d30 msgr2=0x55ccbb4b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.424+0000 7fa6139be740 1 -- 172.21.15.63:0/572121368 shutdown_connections 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.424+0000 7fa6139be740 1 -- 172.21.15.63:0/572121368 wait complete. 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.424+0000 7fa6139be740 1 -- start start 2024-04-25T19:05:47.426 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.425+0000 7fa6139be740 1 Processor -- start 2024-04-25T19:05:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.425+0000 7fa6139be740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa609e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa609e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:45482/0 (socket says 172.21.15.63:45482) 2024-04-25T19:05:47.428 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa609e00640 1 -- 172.21.15.63:0/1721971096 learned_addr learned my addr 172.21.15.63:0/1721971096 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1ae480 con 0x55ccbb4b9140 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1af8f0 con 0x55ccbb4c3a00 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c1aeb50 con 0x55ccbb4b8820 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa6139be740 10 client.0 _ll_register_callbacks cb 0x7fa5f80043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa609400640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.426+0000 7fa60a800640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.429 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 -- 172.21.15.63:0/1721971096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 msgr2=0x55ccbb4cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 -- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 msgr2=0x55ccbb4c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccbb4b4a30 con 0x55ccbb4b9140 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa60a800640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa609400640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c9210 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55ccbb4afe90 tx=0x7fa5f8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a985c5f0770b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.427+0000 7fa5eb400640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f804a030 con 0x55ccbb4b9140 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mon.0 v2:172.21.15.63:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa5f80c1150 con 0x55ccbb4b9140 2024-04-25T19:05:47.430 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5f80d3e50 con 0x55ccbb4b9140 2024-04-25T19:05:47.431 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fa5f80bcc20 con 0x55ccbb4b9140 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa6139be740 10 client.17169 fetch_fsmap learned FSMap version 61 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.428+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=4+}) -- 0x55ccbb4b4a30 con 0x55ccbb4b9140 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mon.0 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5f80d3e50 con 0x55ccbb4b9140 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mon.0 v2:172.21.15.63:3300/0 5 ==== fsmap(e 61) ==== 8354+0+0 (secure 0 0 0) 0x7fa5f80bc180 con 0x55ccbb4b9140 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 10 client.17169 fetch_fsmap finished waiting for FSMap version 61 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 20 client.17169 populate_metadata read hostname 'smithi063' 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb53a5a0 0x55ccbb53c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 4 client.17169 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ccbb42c320 con 0x55ccbb53a5a0 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.429+0000 7fa609e00640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb53a5a0 0x55ccbb53c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.430+0000 7fa5eb400640 10 client.17169 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:47.432 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.430+0000 7fa609e00640 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb53a5a0 0x55ccbb53c9c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841ab30fd95b0407 server_cookie=e3086be7efaa626e in_seq=0 out_seq=0 2024-04-25T19:05:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.779+0000 7fa5eb400640 1 -- 172.21.15.63:0/1721971096 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ccbb42c320 con 0x55ccbb53a5a0 2024-04-25T19:05:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.779+0000 7fa5eb400640 10 client.17169 handle_command_reply: tid=0 2024-04-25T19:05:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.779+0000 7fa6139be740 1 client.17169 shutdown 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.779+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 msgr2=0x55ccbb4c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.779+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c9210 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x55ccbb4afe90 tx=0x7fa5f8005f90 comp rx=0 tx=0).stop 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 shutdown_connections 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55ccbb53a5a0 0x55ccbb53c9c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa5eb400640 0 client.17169 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ccbb4c3a00 0x55ccbb4c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ccbb4b9140 0x55ccbb4c9210 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 --2- 172.21.15.63:0/1721971096 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55ccbb4b8820 0x55ccbb4cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 >> 172.21.15.63:0/1721971096 conn(0x55ccbb4b4d30 msgr2=0x55ccbb4c1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 shutdown_connections 2024-04-25T19:05:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.780+0000 7fa6139be740 1 -- 172.21.15.63:0/1721971096 wait complete. 2024-04-25T19:05:47.782 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.781+0000 7fa6139be740 20 client.17169 trim_cache size 0 max 16384 2024-04-25T19:05:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:05:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:05:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:05:47.783 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 msgr2=0x7fa5f00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 0x7fa5f00a6300 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc009fd0 tx=0x7fa5fc009260 comp rx=0 tx=0).stop 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 msgr2=0x7fa60c1a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 0x7fa60c1a0b80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fa5f802a980 tx=0x7fa5f8030fa0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 shutdown_connections 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa5f00a3e40 0x7fa5f00a6300 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60c10c520 0x7fa60c19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa60c103b00 0x7fa60c1a0b80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.784 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 --2- 172.21.15.63:0/3399273864 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa60c1031e0 0x7fa60c1a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:05:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.782+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 >> 172.21.15.63:0/3399273864 conn(0x7fa60c077fd0 msgr2=0x7fa60c1084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:05:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.783+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 shutdown_connections 2024-04-25T19:05:47.785 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:05:47.783+0000 7fa610c00640 1 -- 172.21.15.63:0/3399273864 wait complete. 2024-04-25T19:05:47.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:47 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:48.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:47 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:48.115 INFO:tasks.workunit.client.0.smithi063.stdout: 10 127302 17.99 MB/sec execute 226 sec latency 236.018 ms 2024-04-25T19:05:48.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:47 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:48 smithi070 ceph-mon[32820]: pgmap v608: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 310 op/s 2024-04-25T19:05:49.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:48 smithi114 ceph-mon[32988]: pgmap v608: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 310 op/s 2024-04-25T19:05:49.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 127884 18.03 MB/sec execute 227 sec latency 118.429 ms 2024-04-25T19:05:49.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:48 smithi063 ceph-mon[28006]: pgmap v608: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.9 MiB/s wr, 310 op/s 2024-04-25T19:05:50.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 128571 18.15 MB/sec execute 228 sec latency 93.354 ms 2024-04-25T19:05:50.930 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:50 smithi063 ceph-mon[28006]: pgmap v609: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.8 MiB/s wr, 595 op/s 2024-04-25T19:05:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:50 smithi070 ceph-mon[32820]: pgmap v609: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.8 MiB/s wr, 595 op/s 2024-04-25T19:05:51.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:50 smithi114 ceph-mon[32988]: pgmap v609: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.8 MiB/s wr, 595 op/s 2024-04-25T19:05:51.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 128913 18.17 MB/sec execute 229 sec latency 273.323 ms 2024-04-25T19:05:52.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 129349 18.17 MB/sec execute 230 sec latency 141.432 ms 2024-04-25T19:05:52.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:52 smithi070 ceph-mon[32820]: pgmap v610: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:52.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:52 smithi070 ceph-mon[32820]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:52 smithi114 ceph-mon[32988]: pgmap v610: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:53.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:52 smithi114 ceph-mon[32988]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:53.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 129890 18.20 MB/sec execute 231 sec latency 103.513 ms 2024-04-25T19:05:53.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:52 smithi063 ceph-mon[28006]: pgmap v610: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:53.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:52 smithi063 ceph-mon[28006]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-04-25T19:05:54.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 130416 18.25 MB/sec execute 232 sec latency 111.736 ms 2024-04-25T19:05:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:54 smithi070 ceph-mon[32820]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:55.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:54 smithi114 ceph-mon[32988]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:55.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 131042 18.31 MB/sec execute 233 sec latency 85.403 ms 2024-04-25T19:05:55.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:54 smithi063 ceph-mon[28006]: pgmap v611: 97 pgs: 97 active+clean; 1.0 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.2 MiB/s wr, 441 op/s 2024-04-25T19:05:56.116 INFO:tasks.workunit.client.0.smithi063.stdout: 10 131500 18.32 MB/sec execute 234 sec latency 309.999 ms 2024-04-25T19:05:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:56 smithi070 ceph-mon[32820]: pgmap v612: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 682 op/s 2024-04-25T19:05:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:56 smithi114 ceph-mon[32988]: pgmap v612: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 682 op/s 2024-04-25T19:05:57.117 INFO:tasks.workunit.client.0.smithi063.stdout: 10 132054 18.40 MB/sec execute 235 sec latency 105.495 ms 2024-04-25T19:05:57.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:56 smithi063 ceph-mon[28006]: pgmap v612: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 11 MiB/s wr, 682 op/s 2024-04-25T19:05:58.118 INFO:tasks.workunit.client.0.smithi063.stdout: 10 132628 18.46 MB/sec execute 236 sec latency 64.439 ms 2024-04-25T19:05:58.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:05:58 smithi070 ceph-mon[32820]: pgmap v613: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 525 op/s 2024-04-25T19:05:59.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:05:58 smithi114 ceph-mon[32988]: pgmap v613: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 525 op/s 2024-04-25T19:05:59.117 INFO:tasks.workunit.client.0.smithi063.stdout: 10 133176 18.50 MB/sec execute 237 sec latency 130.249 ms 2024-04-25T19:05:59.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:05:58 smithi063 ceph-mon[28006]: pgmap v613: 97 pgs: 97 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 525 op/s 2024-04-25T19:06:00.118 INFO:tasks.workunit.client.0.smithi063.stdout: 10 133664 18.53 MB/sec execute 238 sec latency 216.269 ms 2024-04-25T19:06:00.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:00 smithi070 ceph-mon[32820]: pgmap v614: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 746 op/s 2024-04-25T19:06:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.959+0000 7fa6b4800640 1 -- 172.21.15.63:0/1609121889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac102b80 msgr2=0x7fa6ac106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:00.961 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.959+0000 7fa6b4800640 1 --2- 172.21.15.63:0/1609121889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac102b80 0x7fa6ac106bc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009900 tx=0x7fa69c02a3f0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.960+0000 7fa6b4800640 1 -- 172.21.15.63:0/1609121889 shutdown_connections 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.960+0000 7fa6b4800640 1 --2- 172.21.15.63:0/1609121889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac102b80 0x7fa6ac106bc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.960+0000 7fa6b4800640 1 --2- 172.21.15.63:0/1609121889 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1072a0 0x7fa6ac10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.960+0000 7fa6b4800640 1 --2- 172.21.15.63:0/1609121889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac1025b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.960+0000 7fa6b4800640 1 -- 172.21.15.63:0/1609121889 >> 172.21.15.63:0/1609121889 conn(0x7fa6ac0793e0 msgr2=0x7fa6ac0797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.962+0000 7fa6b4800640 1 -- 172.21.15.63:0/1609121889 shutdown_connections 2024-04-25T19:06:00.963 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.962+0000 7fa6b4800640 1 -- 172.21.15.63:0/1609121889 wait complete. 2024-04-25T19:06:00.964 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.963+0000 7fa6b4800640 1 Processor -- start 2024-04-25T19:06:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.963+0000 7fa6b4800640 1 -- start start 2024-04-25T19:06:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac1072a0 0x7fa6ac074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 0x7fa6ac074c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac112230 con 0x7fa6ac0747e0 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1120b0 con 0x7fa6ac1072a0 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.964+0000 7fa6b4800640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1123b0 con 0x7fa6ac1021d0 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54194/0 (socket says 172.21.15.63:54194) 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 -- 172.21.15.63:0/4032361938 learned_addr learned my addr 172.21.15.63:0/4032361938 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1400640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac1072a0 0x7fa6ac074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b2800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 0x7fa6ac074c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 -- 172.21.15.63:0/4032361938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac1072a0 msgr2=0x7fa6ac074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac1072a0 0x7fa6ac074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 -- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 msgr2=0x7fa6ac074c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 0x7fa6ac074c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.967 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b1e00640 1 -- 172.21.15.63:0/4032361938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac1ab370 con 0x7fa6ac1021d0 2024-04-25T19:06:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.965+0000 7fa6b2800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 0x7fa6ac074c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:06:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d950 tx=0x7fa6a000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6a6a00640 1 -- 172.21.15.63:0/4032361938 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0017070 con 0x7fa6ac1021d0 2024-04-25T19:06:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6a6a00640 1 -- 172.21.15.63:0/4032361938 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a0009c30 con 0x7fa6ac1021d0 2024-04-25T19:06:00.968 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6a6a00640 1 -- 172.21.15.63:0/4032361938 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a000f620 con 0x7fa6ac1021d0 2024-04-25T19:06:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ac1ab660 con 0x7fa6ac1021d0 2024-04-25T19:06:00.969 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.966+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ac1abb30 con 0x7fa6ac1021d0 2024-04-25T19:06:00.973 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.972+0000 7fa6b75fc740 1 Processor -- start 2024-04-25T19:06:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.972+0000 7fa6a6a00640 1 -- 172.21.15.63:0/4032361938 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fa6a0010460 con 0x7fa6ac1021d0 2024-04-25T19:06:00.974 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 -- start start 2024-04-25T19:06:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c671c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 0x55733c67edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.975 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55733c5d7900 con 0x55733c67ca00 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55733c57def0 con 0x55733c672140 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.973+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55733c5d38c0 con 0x55733c671820 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c671c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44570/0 (socket says 172.21.15.63:44570) 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 -- 172.21.15.63:0/771851917 learned_addr learned my addr 172.21.15.63:0/771851917 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 -- 172.21.15.63:0/771851917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 msgr2=0x55733c671c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c671c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 -- 172.21.15.63:0/771851917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 msgr2=0x55733c67edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1400640 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 0x55733c67edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 0x55733c67edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.976 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1e00640 1 -- 172.21.15.63:0/771851917 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733c2ea210 con 0x55733c672140 2024-04-25T19:06:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.974+0000 7fa6b1400640 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 0x55733c67edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:06:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.975+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00cda80 tx=0x7fa6a00d6a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65240308a9f7e9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.975+0000 7fa69b000640 1 -- 172.21.15.63:0/771851917 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a00e3070 con 0x55733c672140 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.975+0000 7fa69b000640 1 -- 172.21.15.63:0/771851917 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a00d25a0 con 0x55733c672140 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.975+0000 7fa69b000640 1 -- 172.21.15.63:0/771851917 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a00deab0 con 0x55733c672140 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.975+0000 7fa6a6a00640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 0x7fa6940a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.976+0000 7fa6a6a00640 1 -- 172.21.15.63:0/4032361938 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa6a00d0400 con 0x7fa6ac1021d0 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.976+0000 7fa6b2800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 0x7fa6940a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.976+0000 7fa6b75fc740 1 -- 172.21.15.63:0/771851917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 msgr2=0x55733c67c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.976+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00cda80 tx=0x7fa6a00d6a70 comp rx=0 tx=0).stop 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 -- 172.21.15.63:0/771851917 shutdown_connections 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c67c4c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c67ca00 0x55733c67edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/771851917 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c671c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 -- 172.21.15.63:0/771851917 >> 172.21.15.63:0/771851917 conn(0x55733c66dd30 msgr2=0x55733c66e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:00.979 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 -- 172.21.15.63:0/771851917 shutdown_connections 2024-04-25T19:06:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 -- 172.21.15.63:0/771851917 wait complete. 2024-04-25T19:06:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b2800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 0x7fa6940a6300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8005fd0 tx=0x7fa6a8009500 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.977+0000 7fa6b75fc740 1 -- start start 2024-04-25T19:06:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 Processor -- start 2024-04-25T19:06:00.980 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c679a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c679f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55733c67f630 con 0x55733c671820 2024-04-25T19:06:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55733c591310 con 0x55733c672140 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55733c67f7b0 con 0x55733c67ca00 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b75fc740 10 client.0 _ll_register_callbacks cb 0x55733c55fcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54206/0 (socket says 172.21.15.63:54206) 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.978+0000 7fa6b2800640 1 -- 172.21.15.63:0/1403622026 learned_addr learned my addr 172.21.15.63:0/1403622026 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c679a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b1400640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c679f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 -- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 msgr2=0x55733c679f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c679f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 -- 172.21.15.63:0/1403622026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 msgr2=0x55733c679a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c679a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 -- 172.21.15.63:0/1403622026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733c3b6620 con 0x55733c67ca00 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa6b2800640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80049d0 tx=0x7fa6a8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd4128364c6cf6b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:00.982 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.979+0000 7fa68f400640 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:06:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.980+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8012e60 con 0x55733c67ca00 2024-04-25T19:06:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.980+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa6a800fce0 con 0x55733c67ca00 2024-04-25T19:06:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.980+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6a00d25a0 con 0x55733c67ca00 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.980+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa6a801f420 con 0x55733c67ca00 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.981+0000 7fa6b75fc740 10 client.27433 fetch_fsmap learned FSMap version 62 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.981+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55733c68b920 con 0x55733c67ca00 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.981+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a00d25a0 con 0x55733c67ca00 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.981+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 62) ==== 8362+0+0 (secure 0 0 0) 0x7fa6a80316f0 con 0x55733c67ca00 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.982+0000 7fa6b75fc740 10 client.27433 fetch_fsmap finished waiting for FSMap version 62 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.982+0000 7fa6b75fc740 10 client.27433 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.982+0000 7fa6b75fc740 20 client.27433 populate_metadata read hostname 'smithi063' 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.982+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c68ce30 0x55733c68f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.982+0000 7fa6b75fc740 4 client.27433 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:06:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.983+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55733c5eb8b0 con 0x55733c68ce30 2024-04-25T19:06:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.983+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c68ce30 0x55733c68f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.984+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c68ce30 0x55733c68f260 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6db1ca59b0254ce server_cookie=c9dd175e7ce3cb4 in_seq=0 out_seq=0 2024-04-25T19:06:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.984+0000 7fa68f400640 10 client.27433 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.984+0000 7fa68f400640 1 -- 172.21.15.63:0/1403622026 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fa6a0081ae0 con 0x55733c68ce30 2024-04-25T19:06:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.984+0000 7fa68f400640 10 client.27433 handle_command_reply: tid=0 2024-04-25T19:06:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.984+0000 7fa6b75fc740 1 client.27433 shutdown 2024-04-25T19:06:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 msgr2=0x55733c67a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:00.986 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80049d0 tx=0x7fa6a8004a40 comp rx=0 tx=0).stop 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 shutdown_connections 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c68ce30 0x55733c68f260 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c672140 0x55733c679f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c67ca00 0x55733c67a4c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/1403622026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c679a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 >> 172.21.15.63:0/1403622026 conn(0x55733c66dd30 msgr2=0x55733c673a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa68f400640 0 client.27433 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 shutdown_connections 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.985+0000 7fa6b75fc740 1 -- 172.21.15.63:0/1403622026 wait complete. 2024-04-25T19:06:00.987 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:00.986+0000 7fa6b75fc740 20 client.27433 trim_cache size 0 max 16384 2024-04-25T19:06:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 Processor -- start 2024-04-25T19:06:01.011 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 -- start start 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c6884f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c672140 0x55733c66d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b6ce0 con 0x55733c672140 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b44f0 con 0x55733c67ca00 2024-04-25T19:06:01.012 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.010+0000 7fa6b75fc740 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b2360 con 0x55733c671820 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44598/0 (socket says 172.21.15.63:44598) 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 -- 172.21.15.63:0/3473127016 learned_addr learned my addr 172.21.15.63:0/3473127016 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b1400640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c672140 0x55733c66d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c6884f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 -- 172.21.15.63:0/3473127016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 msgr2=0x55733c6884f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c6884f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 -- 172.21.15.63:0/3473127016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c672140 msgr2=0x55733c66d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c672140 0x55733c66d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b2800640 1 -- 172.21.15.63:0/3473127016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55733c672880 con 0x55733c67ca00 2024-04-25T19:06:01.013 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.011+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c6884f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:06:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.012+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8011dc0 tx=0x7fa6a8005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb01337e2bb56b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.012+0000 7fa69a600640 1 -- 172.21.15.63:0/3473127016 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a803f070 con 0x55733c67ca00 2024-04-25T19:06:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.012+0000 7fa69a600640 1 -- 172.21.15.63:0/3473127016 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6a800fae0 con 0x55733c67ca00 2024-04-25T19:06:01.014 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.012+0000 7fa69a600640 1 -- 172.21.15.63:0/3473127016 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8036040 con 0x55733c67ca00 2024-04-25T19:06:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3473127016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 msgr2=0x55733c694b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8011dc0 tx=0x7fa6a8005c70 comp rx=0 tx=0).stop 2024-04-25T19:06:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3473127016 shutdown_connections 2024-04-25T19:06:01.015 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c694b70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c672140 0x55733c66d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3473127016 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c671820 0x55733c6884f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.014+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3473127016 >> 172.21.15.63:0/3473127016 conn(0x55733c66dd30 msgr2=0x55733c674d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.015+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3473127016 shutdown_connections 2024-04-25T19:06:01.016 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.015+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3473127016 wait complete. 2024-04-25T19:06:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.015+0000 7fa6b75fc740 1 -- start start 2024-04-25T19:06:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.016+0000 7fa6b75fc740 1 Processor -- start 2024-04-25T19:06:01.018 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.017+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.018+0000 7fa6b75fc740 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c672140 0x55733c6821e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.018+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.018+0000 7fa6b1e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47240/0 (socket says 172.21.15.63:47240) 2024-04-25T19:06:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.018+0000 7fa6b1e00640 1 -- 172.21.15.63:0/3564215272 learned_addr learned my addr 172.21.15.63:0/3564215272 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.018+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c6eb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b29b0 con 0x55733c671820 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b3e20 con 0x55733c67ca00 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac1b3080 con 0x55733c672140 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c6eb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b1400640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c672140 0x55733c6821e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 -- 172.21.15.63:0/3564215272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c672140 msgr2=0x55733c6821e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c672140 0x55733c6821e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 -- 172.21.15.63:0/3564215272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 msgr2=0x55733c681ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b75fc740 10 client.0 _ll_register_callbacks cb 0x7fa6a80125c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:06:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.019+0000 7fa6b2800640 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55733c6eb9f0 con 0x55733c67ca00 2024-04-25T19:06:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.020+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6a800fae0 con 0x55733c67ca00 2024-04-25T19:06:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.020+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.020+0000 7fa6b2800640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c6eb3b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8031cd0 tx=0x7fa6a8031d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb09f534a9793dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.020+0000 7fa68f400640 10 client.27428 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa68f400640 1 -- 172.21.15.63:0/3564215272 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8036070 con 0x55733c67ca00 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa68f400640 1 -- 172.21.15.63:0/3564215272 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 8587+0+0 (secure 0 0 0) 0x7fa6a8004980 con 0x55733c67ca00 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa68f400640 1 -- 172.21.15.63:0/3564215272 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7fa6a803f050 con 0x55733c67ca00 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa6b75fc740 10 client.27428 fetch_fsmap learned FSMap version 62 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({fsmap=62}) -- 0x55733c6f2bb0 con 0x55733c67ca00 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.021+0000 7fa68f400640 1 -- 172.21.15.63:0/3564215272 <== mon.2 v2:172.21.15.70:3300/0 4 ==== fsmap(e 62) ==== 8362+0+0 (secure 0 0 0) 0x7fa6a804c030 con 0x55733c67ca00 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 10 client.27428 fetch_fsmap finished waiting for FSMap version 62 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 10 client.27428 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T19:06:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 20 client.27428 populate_metadata read hostname 'smithi063' 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c6f3150 0x55733c6f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 4 client.27428 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55733c5e5320 con 0x55733c6f3150 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.022+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c6f3150 0x55733c6f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.023+0000 7fa6b1e00640 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c6f3150 0x55733c6f5570 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b2109cf9f935824 server_cookie=afc023311f8f00a1 in_seq=0 out_seq=0 2024-04-25T19:06:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.023+0000 7fa68f400640 10 client.27428 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:00 smithi114 ceph-mon[32988]: pgmap v614: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 746 op/s 2024-04-25T19:06:01.118 INFO:tasks.workunit.client.0.smithi063.stdout: 10 133982 18.50 MB/sec execute 239 sec latency 306.410 ms 2024-04-25T19:06:01.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:00 smithi063 ceph-mon[28006]: pgmap v614: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 13 MiB/s wr, 746 op/s 2024-04-25T19:06:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.327+0000 7fa68f400640 1 -- 172.21.15.63:0/3564215272 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55733c5e5320 con 0x55733c6f3150 2024-04-25T19:06:01.329 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.327+0000 7fa68f400640 10 client.27428 handle_command_reply: tid=0 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.327+0000 7fa6b75fc740 1 client.27428 shutdown 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 msgr2=0x55733c6eb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c6eb3b0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8031cd0 tx=0x7fa6a8031d50 comp rx=0 tx=0).stop 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 shutdown_connections 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55733c6f3150 0x55733c6f5570 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55733c67ca00 0x55733c6eb3b0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa68f400640 0 client.27428 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55733c672140 0x55733c6821e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 --2- 172.21.15.63:0/3564215272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55733c671820 0x55733c681ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.328+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 >> 172.21.15.63:0/3564215272 conn(0x55733c66dd30 msgr2=0x55733c6731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.329+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 shutdown_connections 2024-04-25T19:06:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.329+0000 7fa6b75fc740 1 -- 172.21.15.63:0/3564215272 wait complete. 2024-04-25T19:06:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.329+0000 7fa6b75fc740 20 client.27428 trim_cache size 0 max 16384 2024-04-25T19:06:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:{ 2024-04-25T19:06:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "message": "", 2024-04-25T19:06:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout: "return_code": 0 2024-04-25T19:06:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stdout:} 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.331+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 msgr2=0x7fa6940a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.331+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 0x7fa6940a6300 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8005fd0 tx=0x7fa6a8009500 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.331+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 msgr2=0x7fa6ac075e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.331+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000d950 tx=0x7fa6a000de10 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 shutdown_connections 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fa6940a3e40 0x7fa6940a6300 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6ac1072a0 0x7fa6ac074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6ac0747e0 0x7fa6ac074c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 --2- 172.21.15.63:0/4032361938 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fa6ac1021d0 0x7fa6ac075e50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:01.333 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 >> 172.21.15.63:0/4032361938 conn(0x7fa6ac0793e0 msgr2=0x7fa6ac0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 shutdown_connections 2024-04-25T19:06:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi063.front.sepia.ceph.com.smithi063.stderr:2024-04-25T19:06:01.332+0000 7fa6b4800640 1 -- 172.21.15.63:0/4032361938 wait complete. 2024-04-25T19:06:01.558 INFO:tasks.workunit.client.0.smithi063.stdout:[136925] write failed on handle 13547 (Resource temporarily unavailable) 2024-04-25T19:06:01.560 INFO:tasks.workunit.client.0.smithi063.stdout:Child failed with status 1 2024-04-25T19:06:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:06:01.604 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-25T19:06:01.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T19:06:02.733 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7797745e6d8b3ac036504dc61491d368615024da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-25T19:06:02.970 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=667113d0dc44406bac414416df5c2b4e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7797745e6d8b3ac036504dc61491d368615024da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-25T19:06:02.973 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-04-25T19:06:02.983 INFO:tasks.quiescer:joining Quiescers 2024-04-25T19:06:02.983 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-04-25T19:06:02.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:06:02.989 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:02 smithi070 ceph-mon[32820]: pgmap v615: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:02 smithi114 ceph-mon[32988]: pgmap v615: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:03.068 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:02 smithi063 ceph-mon[28006]: pgmap v615: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:03.440 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:04.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:04 smithi063 ceph-mon[28006]: pgmap v616: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:04.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:04 smithi070 ceph-mon[32820]: pgmap v616: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:04 smithi114 ceph-mon[32988]: pgmap v616: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 9.7 MiB/s wr, 461 op/s 2024-04-25T19:06:06.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 -- 172.21.15.63:0/3584511095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc390109a40 msgr2=0x7fc390109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 --2- 172.21.15.63:0/3584511095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc390109a40 0x7fc390109e20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc38000b0a0 tx=0x7fc38000b3b0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 -- 172.21.15.63:0/3584511095 shutdown_connections 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 --2- 172.21.15.63:0/3584511095 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390115b90 0x7fc390117f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 --2- 172.21.15.63:0/3584511095 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc390109a40 0x7fc390109e20 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 --2- 172.21.15.63:0/3584511095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc39010a360 0x7fc390115650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.007+0000 7fc397a00700 1 -- 172.21.15.63:0/3584511095 >> 172.21.15.63:0/3584511095 conn(0x7fc39006dbb0 msgr2=0x7fc39006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:06.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.008+0000 7fc397a00700 1 -- 172.21.15.63:0/3584511095 shutdown_connections 2024-04-25T19:06:06.011 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:05 smithi063 ceph-mon[28006]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T19:06:06.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.009+0000 7fc397a00700 1 -- 172.21.15.63:0/3584511095 wait complete. 2024-04-25T19:06:06.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.010+0000 7fc397a00700 1 Processor -- start 2024-04-25T19:06:06.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.011+0000 7fc397a00700 1 -- start start 2024-04-25T19:06:06.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.011+0000 7fc397a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:06.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.011+0000 7fc397a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:06.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.011+0000 7fc397a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc390115b90 0x7fc39010c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:06.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.011+0000 7fc387e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:06.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc397a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc39011b4d0 con 0x7fc390115b90 2024-04-25T19:06:06.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc397a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc39011b350 con 0x7fc39010a360 2024-04-25T19:06:06.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc397a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7fc39011b650 con 0x7fc390109a40 2024-04-25T19:06:06.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc395000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:06.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc387e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:44618/0 (socket says 172.21.15.63:44618) 2024-04-25T19:06:06.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc395000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:54234/0 (socket says 172.21.15.63:54234) 2024-04-25T19:06:06.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.012+0000 7fc395000700 1 -- 172.21.15.63:0/2776896365 learned_addr learned my addr 172.21.15.63:0/2776896365 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.013+0000 7fc395a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc390115b90 0x7fc39010c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.013+0000 7fc395000700 1 -- 172.21.15.63:0/2776896365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 msgr2=0x7fc3901155a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.014+0000 7fc395000700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.014+0000 7fc395000700 1 -- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc390115b90 msgr2=0x7fc39010c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:06.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.014+0000 7fc395000700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc390115b90 0x7fc39010c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.014+0000 7fc395000700 1 -- 172.21.15.63:0/2776896365 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc39010cab0 con 0x7fc390109a40 2024-04-25T19:06:06.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.014+0000 7fc387e00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:06:06.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.015+0000 7fc395000700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc38000d010 tx=0x7fc380004db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:06.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.016+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380021070 con 0x7fc390109a40 2024-04-25T19:06:06.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.017+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc39010cd40 con 0x7fc390109a40 2024-04-25T19:06:06.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.017+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc390110d30 con 0x7fc390109a40 2024-04-25T19:06:06.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.017+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3800074e0 con 0x7fc390109a40 2024-04-25T19:06:06.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.017+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380004480 con 0x7fc390109a40 2024-04-25T19:06:06.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.019+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc36c005330 con 0x7fc390109a40 2024-04-25T19:06:06.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.023+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7fc380004620 con 0x7fc390109a40 2024-04-25T19:06:06.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.024+0000 7fc38e000700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 0x7fc3740a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:06.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.025+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7fc38000f020 con 0x7fc390109a40 2024-04-25T19:06:06.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.025+0000 7fc387e00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 0x7fc3740a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:06.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.026+0000 7fc387e00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 0x7fc3740a6270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc39010d7e0 tx=0x7fc378009670 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:06.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.030+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc38002a020 con 0x7fc390109a40 2024-04-25T19:06:06.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:05 smithi114 ceph-mon[32988]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T19:06:06.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:05 smithi070 ceph-mon[32820]: osdmap e126: 12 total, 12 up, 12 in 2024-04-25T19:06:06.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.550+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc36c005620 con 0x7fc390109a40 2024-04-25T19:06:06.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.551+0000 7fc38e000700 1 -- 172.21.15.63:0/2776896365 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11874 (secure 0 0 0) 0x7fc38009e890 con 0x7fc390109a40 2024-04-25T19:06:06.557 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:06:06.559 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:05:51.195626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T19:06:06.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.556+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 msgr2=0x7fc3740a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:06.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.556+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 0x7fc3740a6270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc39010d7e0 tx=0x7fc378009670 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.556+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 msgr2=0x7fc390115060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.556+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc38000d010 tx=0x7fc380004db0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 shutdown_connections 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7fc3740a3db0 0x7fc3740a6270 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7fc390109a40 0x7fc390115060 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc390115b90 0x7fc39010c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 --2- 172.21.15.63:0/2776896365 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc39010a360 0x7fc3901155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 >> 172.21.15.63:0/2776896365 conn(0x7fc39006dbb0 msgr2=0x7fc390072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:06.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 shutdown_connections 2024-04-25T19:06:06.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:06.557+0000 7fc397a00700 1 -- 172.21.15.63:0/2776896365 wait complete. 2024-04-25T19:06:06.565 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 62 2024-04-25T19:06:06.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:06 smithi063 ceph-mon[28006]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:06.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:06 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2776896365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:06:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:06 smithi114 ceph-mon[32988]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:06 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2776896365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:06:07.203 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:06 smithi070 ceph-mon[32820]: pgmap v618: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:07.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:06 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2776896365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:06:07.790 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24533 quiesce db --cancel --all' 2024-04-25T19:06:07.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- timeout 75 ceph tell mds.24533 quiesce db --cancel --all 2024-04-25T19:06:08.217 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:08 smithi114 ceph-mon[32988]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:09.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:08 smithi063 ceph-mon[28006]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:09.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:08 smithi070 ceph-mon[32820]: pgmap v619: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.8 MiB/s wr, 556 op/s 2024-04-25T19:06:10.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:09 smithi114 ceph-mon[32988]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:10.127 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:09 smithi063 ceph-mon[28006]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:10.204 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:09 smithi070 ceph-mon[32820]: pgmap v620: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:10.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 -- 172.21.15.63:0/1030703241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f8101030 msgr2=0x7f55f8101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 --2- 172.21.15.63:0/1030703241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f8101030 0x7f55f8101490 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009990 tx=0x7f55e0009ca0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 -- 172.21.15.63:0/1030703241 shutdown_connections 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 --2- 172.21.15.63:0/1030703241 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f81019d0 0x7f55f810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 --2- 172.21.15.63:0/1030703241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8108ad0 0x7f55f8108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 --2- 172.21.15.63:0/1030703241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f8101030 0x7f55f8101490 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 -- 172.21.15.63:0/1030703241 >> 172.21.15.63:0/1030703241 conn(0x7f55f8077fb0 msgr2=0x7f55f80ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 -- 172.21.15.63:0/1030703241 shutdown_connections 2024-04-25T19:06:10.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.737+0000 7f55ff400700 1 -- 172.21.15.63:0/1030703241 wait complete. 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.738+0000 7f55ff400700 1 Processor -- start 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 -- start start 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 0x7f55f8102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f8108ad0 0x7f55f8102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8112be0 con 0x7f55f81019d0 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8112a60 con 0x7f55f8101030 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.739+0000 7f55ff400700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f55f8112d60 con 0x7f55f8108ad0 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35838/0 (socket says 172.21.15.63:35838) 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 -- 172.21.15.63:0/1605597631 learned_addr learned my addr 172.21.15.63:0/1605597631 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55efe00700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 0x7f55f8102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fd400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f8108ad0 0x7f55f8102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 -- 172.21.15.63:0/1605597631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f8108ad0 msgr2=0x7f55f8102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f8108ad0 0x7f55f8102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 -- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 msgr2=0x7f55f8102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 0x7f55f8102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55fca00700 1 -- 172.21.15.63:0/1605597631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f81a7660 con 0x7f55f8101030 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.740+0000 7f55efe00700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 0x7f55f8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:06:10.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55fca00700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e9a0 tx=0x7f55e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55f6000700 1 -- 172.21.15.63:0/1605597631 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e800c9c0 con 0x7f55f8101030 2024-04-25T19:06:10.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f81a7950 con 0x7f55f8101030 2024-04-25T19:06:10.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55f6000700 1 -- 172.21.15.63:0/1605597631 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e800cb60 con 0x7f55f8101030 2024-04-25T19:06:10.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55f6000700 1 -- 172.21.15.63:0/1605597631 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8018680 con 0x7f55f8101030 2024-04-25T19:06:10.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.741+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f81a7eb0 con 0x7f55f8101030 2024-04-25T19:06:10.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.745+0000 7f55f6000700 1 -- 172.21.15.63:0/1605597631 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f55e8010430 con 0x7f55f8101030 2024-04-25T19:06:10.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f5609e45b80 1 Processor -- start 2024-04-25T19:06:10.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f55f6000700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 0x7f55dc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f5609e45b80 1 -- start start 2024-04-25T19:06:10.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f55f6000700 1 -- 172.21.15.63:0/1605597631 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f55e80d0b20 con 0x7f55f8101030 2024-04-25T19:06:10.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f55fd400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 0x7f55dc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.746+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c540021830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f5609e45b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c53ff66f00 con 0x55c5400145d0 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f5609e45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c53ff0d7a0 con 0x55c540013cb0 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f5609e45b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c53ff62ec0 con 0x55c54001f440 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55efe00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35850/0 (socket says 172.21.15.63:35850) 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 -- 172.21.15.63:0/626292362 learned_addr learned my addr 172.21.15.63:0/626292362 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fd400700 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c540021830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 -- 172.21.15.63:0/626292362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 msgr2=0x55c540021830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c540021830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 -- 172.21.15.63:0/626292362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 msgr2=0x55c54001ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.747+0000 7f55fca00700 1 -- 172.21.15.63:0/626292362 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c53ff036f0 con 0x55c540013cb0 2024-04-25T19:06:10.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.748+0000 7f55fd400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 0x7f55dc0a6270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009fd0 tx=0x7f55f000b040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.748+0000 7f55fca00700 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f55e80993c0 tx=0x7f55e809d170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b281347bb737643 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.748+0000 7f55ee000700 1 -- 172.21.15.63:0/626292362 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80a5660 con 0x55c540013cb0 2024-04-25T19:06:10.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.748+0000 7f55ee000700 1 -- 172.21.15.63:0/626292362 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.748+0000 7f55ee000700 1 -- 172.21.15.63:0/626292362 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80a5430 con 0x55c540013cb0 2024-04-25T19:06:10.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 -- 172.21.15.63:0/626292362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 msgr2=0x55c540014090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f55e80993c0 tx=0x7f55e809d170 comp rx=0 tx=0).stop 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 -- 172.21.15.63:0/626292362 shutdown_connections 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c540021830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540014090 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 --2- 172.21.15.63:0/626292362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 -- 172.21.15.63:0/626292362 >> 172.21.15.63:0/626292362 conn(0x55c53ff42fc0 msgr2=0x55c5400101c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 -- 172.21.15.63:0/626292362 shutdown_connections 2024-04-25T19:06:10.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.749+0000 7f5609e45b80 1 -- 172.21.15.63:0/626292362 wait complete. 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 -- start start 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 Processor -- start 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c54001f440 0x55c54001c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c540021e10 con 0x55c5400145d0 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55c53ff208d0 con 0x55c54001f440 2024-04-25T19:06:10.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x55c540021f90 con 0x55c540013cb0 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.750+0000 7f5609e45b80 10 client.0 _ll_register_callbacks cb 0x55c53feff790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57350/0 (socket says 172.21.15.63:57350) 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 -- 172.21.15.63:0/965396463 learned_addr learned my addr 172.21.15.63:0/965396463 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fd400700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c54001f440 0x55c54001c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55efe00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 -- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c54001f440 msgr2=0x55c54001c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c54001f440 0x55c54001c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 -- 172.21.15.63:0/965396463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 msgr2=0x55c54001bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 -- 172.21.15.63:0/965396463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c53feb4940 con 0x55c540013cb0 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55efe00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55fca00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f55e80ce720 tx=0x7f55e80ce110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17e87a4cdfc91b6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.751+0000 7f55e6a00700 10 client.0 ms_handle_connect on v2:172.21.15.114:3300/0 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80b7070 con 0x55c540013cb0 2024-04-25T19:06:10.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mon.1 v2:172.21.15.114:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f55e80a51d0 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f55e8066050 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f5609e45b80 10 client.27454 fetch_fsmap learned FSMap version 62 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c53fefdb60 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mon.1 v2:172.21.15.114:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.752+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mon.1 v2:172.21.15.114:3300/0 5 ==== fsmap(e 62) ==== 8362+0+0 (secure 0 0 0) 0x7f55e80ba030 con 0x55c540013cb0 2024-04-25T19:06:10.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 10 client.27454 fetch_fsmap finished waiting for FSMap version 62 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 10 client.27454 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 20 client.27454 populate_metadata read hostname 'smithi063' 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c54002ea20 0x55c540030e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 4 client.27454 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c53ff746f0 con 0x55c54002ea20 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.753+0000 7f55efe00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c54002ea20 0x55c540030e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.754+0000 7f55efe00700 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c54002ea20 0x55c540030e10 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c131c0ce4e227cd server_cookie=6134c05917aed0ae in_seq=0 out_seq=0 2024-04-25T19:06:10.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.754+0000 7f55e6a00700 10 client.27454 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.754+0000 7f55e6a00700 1 -- 172.21.15.63:0/965396463 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c53ff746f0 con 0x55c54002ea20 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.754+0000 7f55e6a00700 10 client.27454 handle_command_reply: tid=0 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 client.27454 shutdown 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 msgr2=0x55c54001ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f55e80ce720 tx=0x7f55e80ce110 comp rx=0 tx=0).stop 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 shutdown_connections 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c54002ea20 0x55c540030e10 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c540013cb0 0x55c54001ec60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c54001f440 0x55c54001c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 --2- 172.21.15.63:0/965396463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c54001bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f55e6a00700 0 client.27454 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 >> 172.21.15.63:0/965396463 conn(0x55c53ff42fc0 msgr2=0x55c54000fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 shutdown_connections 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 1 -- 172.21.15.63:0/965396463 wait complete. 2024-04-25T19:06:10.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.755+0000 7f5609e45b80 20 client.27454 trim_cache size 0 max 16384 2024-04-25T19:06:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 Processor -- start 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 -- start start 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c540034380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c54000fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81b2020 con 0x55c5400145d0 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81af7f0 con 0x55c540013cb0 2024-04-25T19:06:10.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.777+0000 7f5609e45b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81ad660 con 0x55c54001f440 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:35878/0 (socket says 172.21.15.63:35878) 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 -- 172.21.15.63:0/3539463844 learned_addr learned my addr 172.21.15.63:0/3539463844 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fd400700 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c54000fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55efe00700 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c540034380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 -- 172.21.15.63:0/3539463844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 msgr2=0x55c54000fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c54000fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 -- 172.21.15.63:0/3539463844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 msgr2=0x55c540034380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c540034380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 -- 172.21.15.63:0/3539463844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c540010c90 con 0x55c540013cb0 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55fca00700 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f55e80993c0 tx=0x7f55e809cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82d2ae0b9ff8838f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.778+0000 7f55ed600700 1 -- 172.21.15.63:0/3539463844 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80b7020 con 0x55c540013cb0 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f55ed600700 1 -- 172.21.15.63:0/3539463844 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f55ed600700 1 -- 172.21.15.63:0/3539463844 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80b5030 con 0x55c540013cb0 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 -- 172.21.15.63:0/3539463844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 msgr2=0x55c540033e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f55e80993c0 tx=0x7f55e809cb70 comp rx=0 tx=0).stop 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 -- 172.21.15.63:0/3539463844 shutdown_connections 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c54001f440 0x55c54000fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c540033e40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 --2- 172.21.15.63:0/3539463844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c5400145d0 0x55c540034380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 -- 172.21.15.63:0/3539463844 >> 172.21.15.63:0/3539463844 conn(0x55c53ff42fc0 msgr2=0x55c54000f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 -- 172.21.15.63:0/3539463844 shutdown_connections 2024-04-25T19:06:10.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.779+0000 7f5609e45b80 1 -- 172.21.15.63:0/3539463844 wait complete. 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.780+0000 7f5609e45b80 1 -- start start 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.780+0000 7f5609e45b80 1 Processor -- start 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.780+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c54008f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.780+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5400145d0 0x55c54008f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.780+0000 7f5609e45b80 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 0x55c540089690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f5609e45b80 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81adcb0 con 0x55c54001f440 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f5609e45b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81af120 con 0x55c540013cb0 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f5609e45b80 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f55f81ae380 con 0x55c5400145d0 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f5609e45b80 10 client.0 _ll_register_callbacks cb 0x55c53feff790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 0x55c540089690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fd400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 0x55c540089690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52172/0 (socket says 172.21.15.63:52172) 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fd400700 1 -- 172.21.15.63:0/1492351324 learned_addr learned my addr 172.21.15.63:0/1492351324 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c54008f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55efe00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5400145d0 0x55c54008f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 -- 172.21.15.63:0/1492351324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5400145d0 msgr2=0x55c54008f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5400145d0 0x55c54008f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 -- 172.21.15.63:0/1492351324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 msgr2=0x55c540089690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 0x55c540089690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.781+0000 7f55fca00700 1 -- 172.21.15.63:0/1492351324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5400142e0 con 0x55c540013cb0 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f55fca00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c54008f270 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f55e8010a00 tx=0x7f55e80ce280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1cfd94142441504 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f55e6a00700 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-25T19:06:10.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80b5070 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mon.2 v2:172.21.15.70:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f55e8087140 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) ==== 24+0+0 (secure 0 0 0) 0x7f55e80b7070 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f5609e45b80 10 client.27449 fetch_fsmap learned FSMap version 62 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.782+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=4+}) -- 0x55c5400142e0 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mon.2 v2:172.21.15.70:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55e809acf0 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mon.2 v2:172.21.15.70:3300/0 5 ==== fsmap(e 62) ==== 8362+0+0 (secure 0 0 0) 0x7f55e80ba070 con 0x55c540013cb0 2024-04-25T19:06:10.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 10 client.27449 fetch_fsmap finished waiting for FSMap version 62 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 10 client.27449 resolve_mds: validated gid 24533 aka daemon mds.b 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 20 client.27449 populate_metadata read hostname 'smithi063' 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c540096b10 0x55c540098f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 4 client.27449 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 --> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55c53ff6e790 con 0x55c540096b10 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.783+0000 7f55efe00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c540096b10 0x55c540098f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f55efe00700 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c540096b10 0x55c540098f30 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6db6c2d43c894b1 server_cookie=31d01481e0a529f9 in_seq=0 out_seq=0 2024-04-25T19:06:10.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f55e6a00700 10 client.27449 ms_handle_connect on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f55e6a00700 1 -- 172.21.15.63:0/1492351324 <== mds.0 v2:172.21.15.70:6838/1944035005 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55c53ff6e790 con 0x55c540096b10 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f55e6a00700 10 client.27449 handle_command_reply: tid=0 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 client.27449 shutdown 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 msgr2=0x55c54008f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c54008f270 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f55e8010a00 tx=0x7f55e80ce280 comp rx=0 tx=0).stop 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 shutdown_connections 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:6838/1944035005,v1:172.21.15.70:6839/1944035005] conn(0x55c540096b10 0x55c540098f30 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c54001f440 0x55c540089690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55c540013cb0 0x55c54008f270 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f55e6a00700 0 client.27449 ms_handle_reset on v2:172.21.15.70:6838/1944035005 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 --2- 172.21.15.63:0/1492351324 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x55c5400145d0 0x55c54008f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.784+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 >> 172.21.15.63:0/1492351324 conn(0x55c53ff42fc0 msgr2=0x55c54002af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 shutdown_connections 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f5609e45b80 1 -- 172.21.15.63:0/1492351324 wait complete. 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f5609e45b80 20 client.27449 trim_cache size 0 max 16384 2024-04-25T19:06:10.794 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:10.795 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 62, 2024-04-25T19:06:10.795 INFO:teuthology.orchestra.run.smithi063.stdout: "leader": 24533, 2024-04-25T19:06:10.795 INFO:teuthology.orchestra.run.smithi063.stdout: "set_version": 50, 2024-04-25T19:06:10.795 INFO:teuthology.orchestra.run.smithi063.stdout: "sets": {} 2024-04-25T19:06:10.795 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 msgr2=0x7f55dc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 0x7f55dc0a6270 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009fd0 tx=0x7f55f000b040 comp rx=0 tx=0).stop 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 msgr2=0x7f55f8102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f55e800e9a0 tx=0x7f55e800ecb0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 shutdown_connections 2024-04-25T19:06:10.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f55dc0a3db0 0x7f55dc0a6270 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f55f8108ad0 0x7f55f8102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55f8101030 0x7f55f8102210 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 --2- 172.21.15.63:0/1605597631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55f81019d0 0x7f55f8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.785+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 >> 172.21.15.63:0/1605597631 conn(0x7f55f8077fb0 msgr2=0x7f55f80ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.786+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 shutdown_connections 2024-04-25T19:06:10.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:10.786+0000 7f55ff400700 1 -- 172.21.15.63:0/1605597631 wait complete. 2024-04-25T19:06:11.693 ERROR:teuthology.run_tasks:Manager failed: quiescer Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/quiescer.py", line 411, in task stop_all_quiescers(ctx.ceph[cluster_name].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph_7797745e6d8b3ac036504dc61491d368615024da/qa/tasks/quiescer.py", line 370, in stop_all_quiescers raise RuntimeError(f"error during quiesce thrashing: {thrasher.exception}") RuntimeError: error during quiesce thrashing: Error quiescing set 'e43e1e63': 110 (ETIMEDOUT) 2024-04-25T19:06:11.698 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T19:06:11.709 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T19:06:11.737 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-25T19:06:11.747 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-25T19:06:11.747 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T19:06:11.747 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T19:06:11.747 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-25T19:06:11.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-25T19:06:11.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-25T19:06:12.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 -- 172.21.15.63:0/1513416040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41018f0 msgr2=0x7f0cd4105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/1513416040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41018f0 0x7f0cd4105db0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009860 tx=0x7f0cc402a460 comp rx=0 tx=0).stop 2024-04-25T19:06:12.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 -- 172.21.15.63:0/1513416040 shutdown_connections 2024-04-25T19:06:12.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/1513416040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd41062f0 0x7f0cd4106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/1513416040 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41018f0 0x7f0cd4105db0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/1513416040 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 -- 172.21.15.63:0/1513416040 >> 172.21.15.63:0/1513416040 conn(0x7f0cd4077fd0 msgr2=0x7f0cd40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:12.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 -- 172.21.15.63:0/1513416040 shutdown_connections 2024-04-25T19:06:12.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.215+0000 7f0cdaa00640 1 -- 172.21.15.63:0/1513416040 wait complete. 2024-04-25T19:06:12.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.216+0000 7f0cdaa00640 1 Processor -- start 2024-04-25T19:06:12.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 -- start start 2024-04-25T19:06:12.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 0x7f0cd41a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 0x7f0cd419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd41187e0 con 0x7f0cd4100fd0 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd4118660 con 0x7f0cd41018f0 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cdaa00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd4118960 con 0x7f0cd41062f0 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52200/0 (socket says 172.21.15.63:52200) 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cd3400640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 0x7f0cd41a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.217+0000 7f0cd3e00640 1 -- 172.21.15.63:0/2164922612 learned_addr learned my addr 172.21.15.63:0/2164922612 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd8a00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 0x7f0cd419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 -- 172.21.15.63:0/2164922612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 msgr2=0x7f0cd419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 0x7f0cd419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 -- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 msgr2=0x7f0cd41a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 -- 172.21.15.63:0/2164922612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd419f280 con 0x7f0cd41018f0 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd8a00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 0x7f0cd419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:06:12.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.218+0000 7f0cd3400640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 0x7f0cd41a4e90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4030f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.219+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4040070 con 0x7f0cd41018f0 2024-04-25T19:06:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.219+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd419f510 con 0x7f0cd41018f0 2024-04-25T19:06:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.219+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0cc4005590 con 0x7f0cd41018f0 2024-04-25T19:06:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.219+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4004d30 con 0x7f0cd41018f0 2024-04-25T19:06:12.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.219+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd41ab810 con 0x7f0cd41018f0 2024-04-25T19:06:12.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.220+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd404fce0 con 0x7f0cd41018f0 2024-04-25T19:06:12.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.222+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0cc4039040 con 0x7f0cd41018f0 2024-04-25T19:06:12.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.224+0000 7f0cd0c00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 0x7f0cb80a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.224+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f0cc40f17a0 con 0x7f0cd41018f0 2024-04-25T19:06:12.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.224+0000 7f0cd3e00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 0x7f0cb80a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.225+0000 7f0cd3e00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 0x7f0cb80a6350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0005fd0 tx=0x7f0cc000b4a0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:12.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.228+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0cc40bdd70 con 0x7f0cd41018f0 2024-04-25T19:06:12.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.460+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f0cd41054a0 con 0x7f0cb80a3e90 2024-04-25T19:06:12.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.479+0000 7f0cd0c00640 1 -- 172.21.15.63:0/2164922612 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7f0cd41054a0 con 0x7f0cb80a3e90 2024-04-25T19:06:12.481 INFO:teuthology.orchestra.run.smithi063.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-25T00:00:00", "created": "2024-04-25T18:46:33", "first": "2024-04-25T18:47:00", "last": "2024-04-25T19:06:00", "last_pruned": "2024-04-25T19:06:01", "created_count": 20, "pruned_count": 13, "active": true} 2024-04-25T19:06:12.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 msgr2=0x7f0cb80a6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 0x7f0cb80a6350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0005fd0 tx=0x7f0cc000b4a0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 msgr2=0x7f0cd41a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 0x7f0cd41a4e90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4030f40 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 shutdown_connections 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0cb80a3e90 0x7f0cb80a6350 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0cd41018f0 0x7f0cd41a4e90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0cd41062f0 0x7f0cd419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 --2- 172.21.15.63:0/2164922612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0cd4100fd0 0x7f0cd41a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 >> 172.21.15.63:0/2164922612 conn(0x7f0cd4077fd0 msgr2=0x7f0cd4103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:12.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 shutdown_connections 2024-04-25T19:06:12.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.482+0000 7f0cdaa00640 1 -- 172.21.15.63:0/2164922612 wait complete. 2024-04-25T19:06:12.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-25T19:06:12.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.713+0000 7f87e1200640 1 -- 172.21.15.63:0/525817990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc104ba0 msgr2=0x7f87dc108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.713+0000 7f87e1200640 1 --2- 172.21.15.63:0/525817990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc104ba0 0x7f87dc108be0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009900 tx=0x7f87c402a3f0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 -- 172.21.15.63:0/525817990 shutdown_connections 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 --2- 172.21.15.63:0/525817990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1092c0 0x7f87dc110e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 --2- 172.21.15.63:0/525817990 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc104ba0 0x7f87dc108be0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 --2- 172.21.15.63:0/525817990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc1041f0 0x7f87dc1045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 -- 172.21.15.63:0/525817990 >> 172.21.15.63:0/525817990 conn(0x7f87dc0ffa30 msgr2=0x7f87dc101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:12.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 -- 172.21.15.63:0/525817990 shutdown_connections 2024-04-25T19:06:12.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.714+0000 7f87e1200640 1 -- 172.21.15.63:0/525817990 wait complete. 2024-04-25T19:06:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.715+0000 7f87e1200640 1 Processor -- start 2024-04-25T19:06:12.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.715+0000 7f87e1200640 1 -- start start 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc1092c0 0x7f87dc1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 0x7f87dc19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc114250 con 0x7f87dc1041f0 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1140d0 con 0x7f87dc19a5d0 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87e1200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc1143d0 con 0x7f87dc1092c0 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87db400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52214/0 (socket says 172.21.15.63:52214) 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87db400640 1 -- 172.21.15.63:0/2121596891 learned_addr learned my addr 172.21.15.63:0/2121596891 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:12.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87dbe00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 0x7f87dc19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.716+0000 7f87d3e00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc1092c0 0x7f87dc1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87db400640 1 -- 172.21.15.63:0/2121596891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc1092c0 msgr2=0x7f87dc1a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87db400640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc1092c0 0x7f87dc1a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87db400640 1 -- 172.21.15.63:0/2121596891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 msgr2=0x7f87dc19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87db400640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 0x7f87dc19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87db400640 1 -- 172.21.15.63:0/2121596891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc19b0f0 con 0x7f87dc1041f0 2024-04-25T19:06:12.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.717+0000 7f87dbe00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 0x7f87dc19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:06:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87db400640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e9f0 tx=0x7f87cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:12.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc00cd80 con 0x7f87dc1041f0 2024-04-25T19:06:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc0054d0 con 0x7f87dc1041f0 2024-04-25T19:06:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc19b3e0 con 0x7f87dc1041f0 2024-04-25T19:06:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc010690 con 0x7f87dc1041f0 2024-04-25T19:06:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.718+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc1a75e0 con 0x7f87dc1041f0 2024-04-25T19:06:12.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.720+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc050530 con 0x7f87dc1041f0 2024-04-25T19:06:12.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.722+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f87cc004e90 con 0x7f87dc1041f0 2024-04-25T19:06:12.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.723+0000 7f87d8c00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 0x7f87c00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:12.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.724+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f87cc0d0cd0 con 0x7f87dc1041f0 2024-04-25T19:06:12.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.724+0000 7f87d3e00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 0x7f87c00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:12.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.725+0000 7f87d3e00640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 0x7f87c00a61d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87c4031440 tx=0x7f87c4005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:12.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.729+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87cc09d2a0 con 0x7f87dc1041f0 2024-04-25T19:06:12.770 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:12 smithi114 ceph-mon[32988]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:12.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:12 smithi063 ceph-mon[28006]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:12 smithi070 ceph-mon[32820]: pgmap v621: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:12.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.965+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 --> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f87dc104ba0 con 0x7f87c00a3d10 2024-04-25T19:06:12.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.985+0000 7f87d8c00640 1 -- 172.21.15.63:0/2121596891 <== mgr.25069 v2:172.21.15.63:6800/2505142895 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f87dc104ba0 con 0x7f87c00a3d10 2024-04-25T19:06:12.988 INFO:teuthology.orchestra.run.smithi063.stdout:/ 1m 6m3h 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 msgr2=0x7f87c00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 0x7f87c00a61d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f87c4031440 tx=0x7f87c4005a20 comp rx=0 tx=0).stop 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 msgr2=0x7f87dc1a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e9f0 tx=0x7f87cc00edf0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 shutdown_connections 2024-04-25T19:06:12.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f87c00a3d10 0x7f87c00a61d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87dc19a5d0 0x7f87dc19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f87dc1092c0 0x7f87dc1a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 --2- 172.21.15.63:0/2121596891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f87dc1041f0 0x7f87dc1a0420 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.988+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 >> 172.21.15.63:0/2121596891 conn(0x7f87dc0ffa30 msgr2=0x7f87dc100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.989+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 shutdown_connections 2024-04-25T19:06:12.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:12.989+0000 7f87e1200640 1 -- 172.21.15.63:0/2121596891 wait complete. 2024-04-25T19:06:13.012 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T19:06:13.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:06:13.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 -- 172.21.15.63:0/347241813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0104d50 msgr2=0x7f44a0108d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 --2- 172.21.15.63:0/347241813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0104d50 0x7f44a0108d90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f448c009900 tx=0x7f448c02a400 comp rx=0 tx=0).stop 2024-04-25T19:06:13.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 -- 172.21.15.63:0/347241813 shutdown_connections 2024-04-25T19:06:13.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 --2- 172.21.15.63:0/347241813 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a0109470 0x7f44a0111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 --2- 172.21.15.63:0/347241813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a0104d50 0x7f44a0108d90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 --2- 172.21.15.63:0/347241813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a01043a0 0x7f44a0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 -- 172.21.15.63:0/347241813 >> 172.21.15.63:0/347241813 conn(0x7f44a00ffc00 msgr2=0x7f44a0102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:13.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 -- 172.21.15.63:0/347241813 shutdown_connections 2024-04-25T19:06:13.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.227+0000 7f44a6c00640 1 -- 172.21.15.63:0/347241813 wait complete. 2024-04-25T19:06:13.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.228+0000 7f44a6c00640 1 Processor -- start 2024-04-25T19:06:13.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 -- start start 2024-04-25T19:06:13.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a0109470 0x7f44a019af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 0x7f44a019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0114400 con 0x7f44a01043a0 2024-04-25T19:06:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0114280 con 0x7f44a0109470 2024-04-25T19:06:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a6c00640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f44a0114580 con 0x7f44a019f330 2024-04-25T19:06:13.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f449fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f449fe00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52238/0 (socket says 172.21.15.63:52238) 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f449fe00640 1 -- 172.21.15.63:0/983162141 learned_addr learned my addr 172.21.15.63:0/983162141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f449f400640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a0109470 0x7f44a019af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.229+0000 7f44a4c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 0x7f44a019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 -- 172.21.15.63:0/983162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 msgr2=0x7f44a019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 0x7f44a019f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 -- 172.21.15.63:0/983162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a0109470 msgr2=0x7f44a019af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a0109470 0x7f44a019af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 -- 172.21.15.63:0/983162141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a019fe20 con 0x7f44a01043a0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f44a4c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 0x7f44a019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449fe00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f449000ebc0 tx=0x7f449000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490010630 con 0x7f44a01043a0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4490010c80 con 0x7f44a01043a0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a01a0110 con 0x7f44a01043a0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4490018b60 con 0x7f44a01043a0 2024-04-25T19:06:13.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.230+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a01a7920 con 0x7f44a01043a0 2024-04-25T19:06:13.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.232+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a0068d70 con 0x7f44a01043a0 2024-04-25T19:06:13.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.234+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f4490018d00 con 0x7f44a01043a0 2024-04-25T19:06:13.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.235+0000 7f449cc00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 0x7f44840a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.235+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f44900d0ed0 con 0x7f44a01043a0 2024-04-25T19:06:13.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.235+0000 7f449f400640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 0x7f44840a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.236+0000 7f449f400640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 0x7f44840a6260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f448c031440 tx=0x7f448c005a20 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:13.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.240+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f449009d4a0 con 0x7f44a01043a0 2024-04-25T19:06:13.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.447+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44a010c6a0 con 0x7f44a01043a0 2024-04-25T19:06:13.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.451+0000 7f449cc00640 1 -- 172.21.15.63:0/983162141 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) ==== 74+0+23276 (secure 0 0 0) 0x7f449009cda0 con 0x7f44a01043a0 2024-04-25T19:06:13.454 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:06:13.455 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":126,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T19:06:04.680114+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":34,"snap_epoch":126,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3012688827":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/3560456479":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/2327478809":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6801/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/3611634789":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":34,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":22,"length":1},{"begin":34,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:06:13.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 msgr2=0x7f44840a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 0x7f44840a6260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f448c031440 tx=0x7f448c005a20 comp rx=0 tx=0).stop 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 msgr2=0x7f44a019aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f449000ebc0 tx=0x7f449000c6a0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 shutdown_connections 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f44840a3da0 0x7f44840a6260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44a0109470 0x7f44a019af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f44a019f330 0x7f44a019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 --2- 172.21.15.63:0/983162141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44a01043a0 0x7f44a019aa40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 >> 172.21.15.63:0/983162141 conn(0x7f44a00ffc00 msgr2=0x7f44a0100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:13.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 shutdown_connections 2024-04-25T19:06:13.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.454+0000 7f44a6c00640 1 -- 172.21.15.63:0/983162141 wait complete. 2024-04-25T19:06:13.478 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T18:36:47.142075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T18:42:23.312145+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T18:42:27.756785+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 34, 'snap_epoch': 126, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:06:13.479 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:06:13.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 -- 172.21.15.63:0/3080425558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 msgr2=0x7f7600074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 --2- 172.21.15.63:0/3080425558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f7600074760 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f75f4009900 tx=0x7f75f402a490 comp rx=0 tx=0).stop 2024-04-25T19:06:13.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 -- 172.21.15.63:0/3080425558 shutdown_connections 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 --2- 172.21.15.63:0/3080425558 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760010e610 0x7f7600110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 --2- 172.21.15.63:0/3080425558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7600074ca0 0x7f760010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 --2- 172.21.15.63:0/3080425558 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f7600074760 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 -- 172.21.15.63:0/3080425558 >> 172.21.15.63:0/3080425558 conn(0x7f76000fd920 msgr2=0x7f76000ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 -- 172.21.15.63:0/3080425558 shutdown_connections 2024-04-25T19:06:13.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.688+0000 7f7606400640 1 -- 172.21.15.63:0/3080425558 wait complete. 2024-04-25T19:06:13.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.690+0000 7f7606400640 1 Processor -- start 2024-04-25T19:06:13.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.690+0000 7f7606400640 1 -- start start 2024-04-25T19:06:13.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.690+0000 7f7606400640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7606400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7600074ca0 0x7f76001a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7606400640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760010e610 0x7f760019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7606400640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f76001142a0 con 0x7f760010e610 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7606400640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7600114120 con 0x7f7600074ca0 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7606400640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f7600114420 con 0x7f7600074380 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605a00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760010e610 0x7f760019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:57448/0 (socket says 172.21.15.63:57448) 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 -- 172.21.15.63:0/2128581800 learned_addr learned my addr 172.21.15.63:0/2128581800 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:13.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f75ffe00640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7600074ca0 0x7f76001a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 -- 172.21.15.63:0/2128581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7600074ca0 msgr2=0x7f76001a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7600074ca0 0x7f76001a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 -- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760010e610 msgr2=0x7f760019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760010e610 0x7f760019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.691+0000 7f7605000640 1 -- 172.21.15.63:0/2128581800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760019af20 con 0x7f7600074380 2024-04-25T19:06:13.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f7605000640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75f4005ab0 tx=0x7f75f4003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:13.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f403f070 con 0x7f7600074380 2024-04-25T19:06:13.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75f4031b60 con 0x7f7600074380 2024-04-25T19:06:13.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f760019b1b0 con 0x7f7600074380 2024-04-25T19:06:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4035030 con 0x7f7600074380 2024-04-25T19:06:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.692+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76001a7520 con 0x7f7600074380 2024-04-25T19:06:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.693+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7600075a30 con 0x7f7600074380 2024-04-25T19:06:13.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.695+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f75f4042070 con 0x7f7600074380 2024-04-25T19:06:13.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.697+0000 7f75fd600640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 0x7f75e80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:13.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.697+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f75f40f0e80 con 0x7f7600074380 2024-04-25T19:06:13.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.697+0000 7f75ffe00640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 0x7f75e80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:13.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.698+0000 7f75ffe00640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 0x7f75e80a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0095f0 tx=0x7f75ec005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:13.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.701+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75f40bce00 con 0x7f7600074380 2024-04-25T19:06:13.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:13 smithi114 ceph-mon[32988]: from='client.27455 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:13.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:13 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/983162141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:06:13.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:13 smithi063 ceph-mon[28006]: from='client.27455 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:13.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:13 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/983162141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:06:13.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.908+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f760019b510 con 0x7f7600074380 2024-04-25T19:06:13.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.909+0000 7f75fd600640 1 -- 172.21.15.63:0/2128581800 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) ==== 93+0+10 (secure 0 0 0) 0x7f75f40c1100 con 0x7f7600074380 2024-04-25T19:06:13.912 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-04-25T19:06:13.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.911+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 msgr2=0x7f75e80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.911+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 0x7f75e80a6260 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0095f0 tx=0x7f75ec005f90 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.911+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 msgr2=0x7f76001a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.911+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f75f4005ab0 tx=0x7f75f4003c10 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 shutdown_connections 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f75e80a3da0 0x7f75e80a6260 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7600074ca0 0x7f76001a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760010e610 0x7f760019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 --2- 172.21.15.63:0/2128581800 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f7600074380 0x7f76001a05a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:13.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 >> 172.21.15.63:0/2128581800 conn(0x7f76000fd920 msgr2=0x7f76000ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:13.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 shutdown_connections 2024-04-25T19:06:13.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:13.912+0000 7f7606400640 1 -- 172.21.15.63:0/2128581800 wait complete. 2024-04-25T19:06:13.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:06:13.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:13 smithi070 ceph-mon[32820]: from='client.27455 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:13.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:13 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/983162141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:06:14.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 -- 172.21.15.63:0/3191641602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c60102060 msgr2=0x7f0c60102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 --2- 172.21.15.63:0/3191641602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c60102060 0x7f0c60102440 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009900 tx=0x7f0c5002a410 comp rx=0 tx=0).stop 2024-04-25T19:06:14.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 -- 172.21.15.63:0/3191641602 shutdown_connections 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 --2- 172.21.15.63:0/3191641602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c6010ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 --2- 172.21.15.63:0/3191641602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102a10 0x7f0c60106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 --2- 172.21.15.63:0/3191641602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c60102060 0x7f0c60102440 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 -- 172.21.15.63:0/3191641602 >> 172.21.15.63:0/3191641602 conn(0x7f0c600fd920 msgr2=0x7f0c600ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 -- 172.21.15.63:0/3191641602 shutdown_connections 2024-04-25T19:06:14.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.145+0000 7f0c65600640 1 -- 172.21.15.63:0/3191641602 wait complete. 2024-04-25T19:06:14.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.146+0000 7f0c65600640 1 Processor -- start 2024-04-25T19:06:14.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 -- start start 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 0x7f0c601a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c601a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 0x7f0c6019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0c60112090 con 0x7f0c6019a750 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c60111f10 con 0x7f0c60107130 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c65600640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f0c60112210 con 0x7f0c60102060 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c64c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 0x7f0c6019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c5fe00640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 0x7f0c601a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c64c00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 0x7f0c6019abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52262/0 (socket says 172.21.15.63:52262) 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c5f400640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c601a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.147+0000 7f0c64c00640 1 -- 172.21.15.63:0/1452133117 learned_addr learned my addr 172.21.15.63:0/1452133117 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 -- 172.21.15.63:0/1452133117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 msgr2=0x7f0c601a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c601a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 -- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 msgr2=0x7f0c6019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 0x7f0c6019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 -- 172.21.15.63:0/1452133117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6019b350 con 0x7f0c60102060 2024-04-25T19:06:14.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5f400640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c601a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5fe00640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 0x7f0c601a05f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002a930 tx=0x7f0c50031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.148+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5003f030 con 0x7f0c60102060 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.149+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c50004490 con 0x7f0c60102060 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.149+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6019b5e0 con 0x7f0c60102060 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.149+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50035070 con 0x7f0c60102060 2024-04-25T19:06:14.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.149+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c601a7680 con 0x7f0c60102060 2024-04-25T19:06:14.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.150+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c60103530 con 0x7f0c60102060 2024-04-25T19:06:14.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.152+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f0c50042030 con 0x7f0c60102060 2024-04-25T19:06:14.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.154+0000 7f0c5cc00640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 0x7f0c440a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.154+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 8631+0+0 (secure 0 0 0) 0x7f0c500f0d90 con 0x7f0c60102060 2024-04-25T19:06:14.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.154+0000 7f0c5f400640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 0x7f0c440a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.155+0000 7f0c5f400640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 0x7f0c440a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0c6019bfa0 tx=0x7f0c4c005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:14.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.156+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c500bcd10 con 0x7f0c60102060 2024-04-25T19:06:14.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.368+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0c60102a10 con 0x7f0c60102060 2024-04-25T19:06:14.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.369+0000 7f0c5cc00640 1 -- 172.21.15.63:0/1452133117 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) ==== 104+0+11 (secure 0 0 0) 0x7f0c500c1010 con 0x7f0c60102060 2024-04-25T19:06:14.371 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 32 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 msgr2=0x7f0c440a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 0x7f0c440a61e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0c6019bfa0 tx=0x7f0c4c005f90 comp rx=0 tx=0).stop 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 msgr2=0x7f0c601a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 0x7f0c601a05f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0c5002a930 tx=0x7f0c50031a20 comp rx=0 tx=0).stop 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 shutdown_connections 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f0c440a3d20 0x7f0c440a61e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c60107130 0x7f0c601a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c6019a750 0x7f0c6019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 --2- 172.21.15.63:0/1452133117 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f0c60102060 0x7f0c601a05f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 >> 172.21.15.63:0/1452133117 conn(0x7f0c600fd920 msgr2=0x7f0c600fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:14.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 shutdown_connections 2024-04-25T19:06:14.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.373+0000 7f0c65600640 1 -- 172.21.15.63:0/1452133117 wait complete. 2024-04-25T19:06:14.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:06:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 -- 172.21.15.63:0/314612087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 msgr2=0x7f5ecc10bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 --2- 172.21.15.63:0/314612087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc10bdb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4009860 tx=0x7f5eb402a3d0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 -- 172.21.15.63:0/314612087 shutdown_connections 2024-04-25T19:06:14.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 --2- 172.21.15.63:0/314612087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc10bdb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 --2- 172.21.15.63:0/314612087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc10e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 --2- 172.21.15.63:0/314612087 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 0x7f5ecc103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.616+0000 7f5ecb200640 1 -- 172.21.15.63:0/314612087 >> 172.21.15.63:0/314612087 conn(0x7f5ecc0fc7d0 msgr2=0x7f5ecc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.617+0000 7f5ecb200640 1 -- 172.21.15.63:0/314612087 shutdown_connections 2024-04-25T19:06:14.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.617+0000 7f5ecb200640 1 -- 172.21.15.63:0/314612087 wait complete. 2024-04-25T19:06:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 Processor -- start 2024-04-25T19:06:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 -- start start 2024-04-25T19:06:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 0x7f5ecc1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc1a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc1120b0 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc111f30 con 0x7f5ecc1038d0 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ecb200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f5ecc112230 con 0x7f5ecc102fb0 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5eca800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5eca800640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52282/0 (socket says 172.21.15.63:52282) 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5eca800640 1 -- 172.21.15.63:0/3882722610 learned_addr learned my addr 172.21.15.63:0/3882722610 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:14.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.618+0000 7f5ec9e00640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 0x7f5ecc1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5ec1400640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc1a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 -- 172.21.15.63:0/3882722610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 msgr2=0x7f5ecc1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 0x7f5ecc1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 -- 172.21.15.63:0/3882722610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 msgr2=0x7f5ecc1a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc1a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 -- 172.21.15.63:0/3882722610 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ecc19af90 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5ec1400640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5eca800640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400efc0 tx=0x7f5ec400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:14.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.619+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4003970 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.620+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ecc19b280 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.620+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ec4004020 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.620+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec400b940 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.620+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ecc1a7540 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.621+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ecc0699c0 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.623+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5ec401d020 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.624+0000 7f5ec3400640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 0x7f5eb00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:14.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.625+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8587+0+0 (secure 0 0 0) 0x7f5ec40d0760 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.625+0000 7f5ec9e00640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 0x7f5eb00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:14.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.626+0000 7f5ec9e00640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 0x7f5eb00a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009bd0 tx=0x7f5ebc005f90 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:14.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.629+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ec409cd60 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:14 smithi114 ceph-mon[32988]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:14.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:14 smithi114 ceph-mon[32988]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:14 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2128581800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:06:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:14 smithi114 ceph-mon[32988]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T19:06:14.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:14 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1452133117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:06:14.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.836+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5ecc19b540 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.837+0000 7f5ec3400640 1 -- 172.21.15.63:0/3882722610 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) ==== 100+0+11 (secure 0 0 0) 0x7f5ec409c660 con 0x7f5ecc10c3d0 2024-04-25T19:06:14.840 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 64 2024-04-25T19:06:14.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.840+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 msgr2=0x7f5eb00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.840+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 0x7f5eb00a61e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009bd0 tx=0x7f5ebc005f90 comp rx=0 tx=0).stop 2024-04-25T19:06:14.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.840+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 msgr2=0x7f5ecc19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:14.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.840+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400efc0 tx=0x7f5ec400c800 comp rx=0 tx=0).stop 2024-04-25T19:06:14.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 shutdown_connections 2024-04-25T19:06:14.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f5eb00a3d20 0x7f5eb00a61e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5ecc1038d0 0x7f5ecc1a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ecc10c3d0 0x7f5ecc19a6d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 --2- 172.21.15.63:0/3882722610 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5ecc102fb0 0x7f5ecc1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 >> 172.21.15.63:0/3882722610 conn(0x7f5ecc0fc7d0 msgr2=0x7f5ecc101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 shutdown_connections 2024-04-25T19:06:14.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:14.841+0000 7f5ecb200640 1 -- 172.21.15.63:0/3882722610 wait complete. 2024-04-25T19:06:14.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:14 smithi063 ceph-mon[28006]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:14.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:14 smithi063 ceph-mon[28006]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:14.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:14 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2128581800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:06:14.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:14 smithi063 ceph-mon[28006]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T19:06:14.861 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:14 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1452133117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:06:14.864 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T19:06:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:14 smithi070 ceph-mon[32820]: from='client.17214 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T19:06:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:14 smithi070 ceph-mon[32820]: pgmap v622: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.0 MiB/s wr, 374 op/s 2024-04-25T19:06:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:14 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2128581800' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T19:06:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:14 smithi070 ceph-mon[32820]: osdmap e127: 12 total, 12 up, 12 in 2024-04-25T19:06:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:14 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1452133117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T19:06:15.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.073+0000 7f8093200640 1 -- 172.21.15.63:0/3611921959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 msgr2=0x7f8094069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.073+0000 7f8093200640 1 --2- 172.21.15.63:0/3611921959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f8094069620 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f807c009be0 tx=0x7f807c02a6c0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 -- 172.21.15.63:0/3611921959 shutdown_connections 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 --2- 172.21.15.63:0/3611921959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f8094069620 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 --2- 172.21.15.63:0/3611921959 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f80941047e0 0x7f8094104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 --2- 172.21.15.63:0/3611921959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8094106c90 0x7f80940690e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 -- 172.21.15.63:0/3611921959 >> 172.21.15.63:0/3611921959 conn(0x7f8094076790 msgr2=0x7f8094076ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:15.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 -- 172.21.15.63:0/3611921959 shutdown_connections 2024-04-25T19:06:15.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.074+0000 7f8093200640 1 -- 172.21.15.63:0/3611921959 wait complete. 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.075+0000 7f8093200640 1 Processor -- start 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.075+0000 7f8093200640 1 -- start start 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8094106c90 0x7f80941a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f809419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f80941184f0 con 0x7f80941047e0 2024-04-25T19:06:15.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8094118370 con 0x7f8094107100 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8093200640 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f8094118670 con 0x7f8094106c90 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8091e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8091e00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52294/0 (socket says 172.21.15.63:52294) 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8091e00640 1 -- 172.21.15.63:0/2395324050 learned_addr learned my addr 172.21.15.63:0/2395324050 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8092800640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f809419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.076+0000 7f8091400640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8094106c90 0x7f80941a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 -- 172.21.15.63:0/2395324050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8094106c90 msgr2=0x7f80941a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8094106c90 0x7f80941a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 -- 172.21.15.63:0/2395324050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 msgr2=0x7f809419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f809419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 -- 172.21.15.63:0/2395324050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809419f0c0 con 0x7f80941047e0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8092800640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f809419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8091e00640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f808400e9c0 tx=0x7f808400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808400cde0 con 0x7f80941047e0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809419f3b0 con 0x7f80941047e0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.077+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80840035f0 con 0x7f80941047e0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.078+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084018630 con 0x7f80941047e0 2024-04-25T19:06:15.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.078+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80941ab650 con 0x7f80941047e0 2024-04-25T19:06:15.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.079+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8094068c90 con 0x7f80941047e0 2024-04-25T19:06:15.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.081+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f8084004000 con 0x7f80941047e0 2024-04-25T19:06:15.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.082+0000 7f808aa00640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 0x7f80740a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:15.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.082+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8587+0+0 (secure 0 0 0) 0x7f80840d0a90 con 0x7f80941047e0 2024-04-25T19:06:15.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.083+0000 7f8091400640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 0x7f80740a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:15.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.084+0000 7f8091400640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 0x7f80740a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f80 tx=0x7f807c005fd0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:15.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.087+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8084009510 con 0x7f80941047e0 2024-04-25T19:06:15.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.352+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f809419fe70 con 0x7f80941047e0 2024-04-25T19:06:15.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.365+0000 7f808aa00640 1 -- 172.21.15.63:0/2395324050 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746061 (secure 0 0 0) 0x7f808409d090 con 0x7f80941047e0 2024-04-25T19:06:15.371 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 26, 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout: "active_gid": 25069, 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout: "active_name": "x", 2024-04-25T19:06:15.372 INFO:teuthology.orchestra.run.smithi063.stdout: "active_addrs": { 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6800", 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2505142895 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-25T19:06:15.373 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6801", 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2505142895 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: "active_addr": "172.21.15.63:6801/2505142895", 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: "active_change": "2024-04-25T18:46:32.881393+0000", 2024-04-25T19:06:15.374 INFO:teuthology.orchestra.run.smithi063.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "available": true, 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "standbys": [ 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "gid": 25049, 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "y", 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_features": 4540701547738038271, 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: "available_modules": [ 2024-04-25T19:06:15.375 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "alerts", 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.376 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.377 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.378 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.379 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.380 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.381 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.382 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.383 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.385 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_destination": { 2024-04-25T19:06:15.386 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_destination", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.387 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_from_name": { 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_from_name", 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.388 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "Ceph", 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email From: name", 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.389 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_host": { 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_host", 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.390 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP server", 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.391 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_password": { 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_password", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.392 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Password to authenticate with", 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_port": { 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_port", 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.393 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "465", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP port", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.394 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_sender": { 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_sender", 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.395 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.396 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_ssl": { 2024-04-25T19:06:15.397 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_ssl", 2024-04-25T19:06:15.397 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.397 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.397 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.397 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.398 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_user": { 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_user", 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.399 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "User to authenticate as", 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "balancer", 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.401 INFO:teuthology.orchestra.run.smithi063.stdout: "active": { 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "active", 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.402 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.403 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_time": { 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_time", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0000", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.404 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_weekday": { 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_weekday", 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:15.405 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:15.406 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_max_iterations": { 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.407 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "25", 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "250", 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.408 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_metrics": { 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_metrics", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.409 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_step": { 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_step", 2024-04-25T19:06:15.410 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.5", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0.001", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "0.999", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:06:15.411 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "end_time": { 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_time", 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.412 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2359", 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.413 INFO:teuthology.orchestra.run.smithi063.stdout: "end_weekday": { 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_weekday", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:15.414 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.415 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.416 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.417 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.418 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.419 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.420 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.421 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.422 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "min_score": { 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_score", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.424 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.425 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "upmap", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-compat", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "none", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "read", 2024-04-25T19:06:15.426 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap", 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap-read" 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Balancer mode", 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.427 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_ids": { 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_ids", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.428 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:15.429 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:06:15.430 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_deviation": { 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_deviation", 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.431 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.432 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_optimizations": { 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.433 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.434 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm", 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_down_multiplier": { 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_down_multiplier", 2024-04-25T19:06:15.435 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:06:15.436 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_refresh_rate": { 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_refresh_rate", 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.437 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "20", 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_starting_port": { 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_starting_port", 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4721", 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.439 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_ptrace": { 2024-04-25T19:06:15.440 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_ptrace", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:06:15.441 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_interval": { 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_interval", 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.442 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.7", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.444 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm_log_destination": { 2024-04-25T19:06:15.445 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm_log_destination", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.446 INFO:teuthology.orchestra.run.smithi063.stdout: "file", 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "file,syslog", 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "syslog" 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.447 INFO:teuthology.orchestra.run.smithi063.stdout: "cgroups_split": { 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cgroups_split", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.448 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "config_checks_enabled": { 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_checks_enabled", 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.449 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.450 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "config_dashboard": { 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_dashboard", 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.451 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.452 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_alertmanager": { 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_alertmanager", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.453 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_base": { 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_base", 2024-04-25T19:06:15.454 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:06:15.455 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_elasticsearch": { 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.456 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "elasticsearch container image", 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_grafana": { 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_grafana", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.458 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_haproxy": { 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_haproxy", 2024-04-25T19:06:15.459 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "HAproxy container image", 2024-04-25T19:06:15.460 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_agent": { 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.461 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.462 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_collector": { 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:06:15.463 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_query": { 2024-04-25T19:06:15.464 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger query container image", 2024-04-25T19:06:15.465 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_keepalived": { 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_keepalived", 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Keepalived container image", 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.467 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_loki": { 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_loki", 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:06:15.468 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Loki container image", 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_node_exporter": { 2024-04-25T19:06:15.469 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_node_exporter", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.470 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_nvmeof": { 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_nvmeof", 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.471 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Nvme-of container image", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.472 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_prometheus": { 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_prometheus", 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:06:15.473 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.474 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_promtail": { 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_promtail", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.475 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Promtail container image", 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_samba": { 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_samba", 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.476 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.477 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_snmp_gateway": { 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:06:15.478 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "container_init": { 2024-04-25T19:06:15.479 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_init", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.480 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "daemon_cache_timeout": { 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.481 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.482 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "900", 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.483 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "default_registry": { 2024-04-25T19:06:15.484 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_registry", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.485 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "device_cache_timeout": { 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_cache_timeout", 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.486 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1800", 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.487 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "device_enhanced_scan": { 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_enhanced_scan", 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.488 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "facts_cache_timeout": { 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "facts_cache_timeout", 2024-04-25T19:06:15.489 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:06:15.490 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "grafana_dashboards_path": { 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.491 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "host_check_interval": { 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "host_check_interval", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.493 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "hw_monitoring": { 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hw_monitoring", 2024-04-25T19:06:15.494 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:06:15.495 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "inventory_list_all": { 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "inventory_list_all", 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.496 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.498 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.499 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "log_refresh_metadata": { 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_refresh_metadata", 2024-04-25T19:06:15.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.501 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.502 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.503 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.504 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.505 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.506 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.507 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.508 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.509 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "*", 2024-04-25T19:06:15.510 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "max_count_per_host": { 2024-04-25T19:06:15.511 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_count_per_host", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:06:15.512 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd_draining_count": { 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_osd_draining_count", 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.513 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.514 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "migration_current": { 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "migration_current", 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "internal - do not modify", 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:15.516 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "root", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm-package", 2024-04-25T19:06:15.517 INFO:teuthology.orchestra.run.smithi063.stdout: "root" 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "oob_default_addr": { 2024-04-25T19:06:15.518 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "oob_default_addr", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "169.254.1.1", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.519 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "prometheus_alerts_path": { 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.520 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.521 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_insecure": { 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_insecure", 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.522 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.523 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_password": { 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_password", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.524 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_url": { 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_url", 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.525 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.526 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_username": { 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_username", 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.527 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.528 INFO:teuthology.orchestra.run.smithi063.stdout: "secure_monitoring_stack": { 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.529 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "service_discovery_port": { 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "service_discovery_port", 2024-04-25T19:06:15.530 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8765", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.531 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "ssh_config_file": { 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssh_config_file", 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.532 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.533 INFO:teuthology.orchestra.run.smithi063.stdout: "use_agent": { 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_agent", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.534 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "use_repo_digest": { 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_repo_digest", 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.535 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.536 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_failed_host_check": { 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.537 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.538 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_daemons": { 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.539 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_hosts": { 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.540 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.541 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crash", 2024-04-25T19:06:15.542 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.547 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.547 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.548 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.549 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.550 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.551 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.552 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.553 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.555 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "retain_interval": { 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retain_interval", 2024-04-25T19:06:15.556 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "31536000", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:06:15.557 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_recent_interval": { 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_recent_interval", 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.558 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1209600", 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dashboard", 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.560 INFO:teuthology.orchestra.run.smithi063.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.561 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:06:15.562 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.563 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.564 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.566 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:06:15.567 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.568 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.569 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.571 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:06:15.572 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.573 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.574 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.575 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.576 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:06:15.577 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.578 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.579 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.580 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.581 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:06:15.582 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.583 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.584 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.585 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.586 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_URL": { 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.588 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:06:15.589 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.590 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.591 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.593 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.594 INFO:teuthology.orchestra.run.smithi063.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.595 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.596 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.597 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.598 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "[]", 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.599 INFO:teuthology.orchestra.run.smithi063.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "{}", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.600 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.601 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.602 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.603 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.604 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.605 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:06:15.606 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.607 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.608 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.609 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.610 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:06:15.611 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.612 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.613 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.615 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:06:15.616 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.617 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.618 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.619 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.620 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.621 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.622 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.623 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "45", 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.625 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:06:15.626 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.627 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.628 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.629 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.630 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:06:15.631 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.632 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:15.633 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.634 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.635 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:06:15.636 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.637 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "cross_origin_url": { 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cross_origin_url", 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "crt_file": { 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crt_file", 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.641 INFO:teuthology.orchestra.run.smithi063.stdout: "debug": { 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "debug", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.642 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "jwt_token_ttl": { 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "jwt_token_ttl", 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.643 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "28800", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.644 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.645 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.646 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.647 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.647 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.647 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.649 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.649 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.649 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.649 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.649 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.650 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.651 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.653 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.654 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.655 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.656 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "motd": { 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "motd", 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.657 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "The message of the day", 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.658 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.659 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.660 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.661 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.662 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8080", 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.664 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl_server_port": { 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl_server_port", 2024-04-25T19:06:15.665 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8443", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.666 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.667 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "redirect", 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect" 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.668 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.669 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.670 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "url_prefix": { 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url_prefix", 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.671 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.672 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "devicehealth", 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_monitoring": { 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_monitoring", 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.673 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "monitor device health metrics", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.674 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.675 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.676 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.677 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.678 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.679 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.680 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.681 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.682 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "mark_out_threshold": { 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mark_out_threshold", 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.683 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2419200", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.684 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": { 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_name", 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.685 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "device_health_metrics", 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "retention_period": { 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retention_period", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15552000", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.687 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_frequency": { 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_frequency", 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.688 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "self_heal": { 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "self_heal", 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.690 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:15.691 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.692 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_threshold": { 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_threshold", 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7257600", 2024-04-25T19:06:15.693 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.694 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "diskprediction_local", 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.695 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.696 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.697 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.698 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.699 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.700 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.701 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.702 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.703 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "predict_interval": { 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predict_interval", 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:15.704 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "predictor_model": { 2024-04-25T19:06:15.705 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predictor_model", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "prophetstor", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.706 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:15.707 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.708 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "influx", 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": false, 2024-04-25T19:06:15.709 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "batch_size": { 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "batch_size", 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5000", 2024-04-25T19:06:15.710 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.711 INFO:teuthology.orchestra.run.smithi063.stdout: "database": { 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "database", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "ceph", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.712 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "hostname": { 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hostname", 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.713 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.714 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "30", 2024-04-25T19:06:15.715 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "5", 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.716 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.717 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.718 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.719 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.721 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.722 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.723 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.724 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "password": { 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "password", 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.725 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "port": { 2024-04-25T19:06:15.726 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "port", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8086", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server port", 2024-04-25T19:06:15.727 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.728 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "false", 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.729 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "threads": { 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threads", 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:15.730 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "32", 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "username": { 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "username", 2024-04-25T19:06:15.731 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:06:15.732 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "verify_ssl": { 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "verify_ssl", 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.733 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "true", 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "insights", 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.735 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.736 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.737 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.738 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.739 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.740 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.741 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.742 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "iostat", 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.744 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.745 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.746 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.747 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.748 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.749 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.750 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.751 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "k8sevents", 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_event_retention_days": { 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.753 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7", 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.754 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "config_check_secs": { 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_check_secs", 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "10", 2024-04-25T19:06:15.755 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.756 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.757 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.758 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.759 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.760 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.761 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.762 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.763 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.764 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "localpool", 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "failure_domain": { 2024-04-25T19:06:15.765 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "failure_domain", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "host", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.766 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.767 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.768 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.770 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.771 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.772 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.773 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.774 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.775 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": { 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_size", 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.776 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "num_rep": { 2024-04-25T19:06:15.777 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "num_rep", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.778 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": { 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_num", 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.779 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "128", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.780 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "prefix": { 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prefix", 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.781 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "subtree": { 2024-04-25T19:06:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "subtree", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "rack", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.783 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.784 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mds_autoscaler", 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.785 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.786 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.787 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.788 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.789 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.790 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.791 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.792 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirroring", 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.794 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.795 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.796 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.797 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.798 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.799 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.800 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.801 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.802 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.803 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "nfs", 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.804 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.805 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.806 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.807 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.808 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.809 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.810 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.811 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:15.812 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "fail_fs": { 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "fail_fs", 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.813 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.814 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.815 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.816 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.817 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.818 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.819 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.820 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.821 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.822 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator": { 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.823 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm", 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "rook", 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: "test_orchestrator" 2024-04-25T19:06:15.824 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Orchestrator backend", 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.825 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_perf_query", 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.826 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.827 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.828 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.829 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.830 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.831 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.832 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.833 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_support", 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.834 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.835 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.836 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.837 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.838 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.839 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.840 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.841 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.842 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_autoscaler", 2024-04-25T19:06:15.843 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.844 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.845 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.846 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.847 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.848 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.849 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.850 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.851 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:15.852 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.853 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "threshold": { 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threshold", 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.854 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1.0", 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "scaling threshold", 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.855 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "progress", 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.857 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_pg_recovery_event": { 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.858 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:15.859 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.860 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.861 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.862 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.863 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.864 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.865 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.866 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.867 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.868 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "max_completed_events": { 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_completed_events", 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "50", 2024-04-25T19:06:15.869 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.870 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.871 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.872 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus", 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "cache": { 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cache", 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.873 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.874 INFO:teuthology.orchestra.run.smithi063.stdout: "exclude_perf_counters": { 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "exclude_perf_counters", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.875 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.876 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.877 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.878 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.879 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.880 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.881 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.882 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.883 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools": { 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools", 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.885 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:06:15.886 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "300", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.887 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_interval": { 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_interval", 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.888 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15.0", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.889 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:15.890 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.891 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "9283", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.892 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "stale_cache_strategy": { 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stale_cache_strategy", 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.893 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "log", 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.894 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.895 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "default", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "default", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "error" 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.896 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.897 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.898 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_support", 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.899 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.900 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.901 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.902 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.903 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.904 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.905 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.906 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.907 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_snap_create": { 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:15.908 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.909 INFO:teuthology.orchestra.run.smithi063.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.910 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: "trash_purge_schedule": { 2024-04-25T19:06:15.911 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "trash_purge_schedule", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.912 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "restful", 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.913 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_auth": { 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_auth", 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.914 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:15.915 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.916 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.917 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.918 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.919 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.920 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.921 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.922 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.923 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.924 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:15.925 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.926 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.927 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.928 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rgw", 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.929 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.930 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.931 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.932 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.934 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.935 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.936 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.937 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rook", 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.938 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.939 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.940 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.941 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.942 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.943 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.944 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.945 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.946 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_class": { 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "storage_class", 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.947 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "local", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.948 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "selftest", 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.949 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.950 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.951 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.952 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.953 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.954 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.955 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.956 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.957 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "roption1": { 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption1", 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.958 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.959 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "roption2": { 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption2", 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "xyz", 2024-04-25T19:06:15.960 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.961 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption1": { 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption1", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.962 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption2": { 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption2", 2024-04-25T19:06:15.963 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.964 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption3": { 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption3", 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.965 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption4": { 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption4", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.967 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption5": { 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption5", 2024-04-25T19:06:15.968 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.969 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption6": { 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption6", 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.970 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption7": { 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption7", 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:15.972 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "42", 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "testkey": { 2024-04-25T19:06:15.973 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testkey", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.974 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "testlkey": { 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testlkey", 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.975 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.976 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "testnewline": { 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testnewline", 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.977 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snap_schedule", 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_m_granularity": { 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_m_granularity", 2024-04-25T19:06:15.979 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:06:15.980 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "dump_on_update": { 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dump_on_update", 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.981 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.982 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.983 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.984 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.985 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.986 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.987 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.988 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.989 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.990 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stats", 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:15.991 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.992 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.993 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:15.994 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.995 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.996 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:15.997 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:15.998 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:15.999 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.000 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "status", 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.001 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.002 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.004 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.005 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.006 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.008 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.009 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telegraf", 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "address": { 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "address", 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.010 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.011 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15", 2024-04-25T19:06:16.012 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.014 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.015 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.016 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.017 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.018 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.019 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.020 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.021 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telemetry", 2024-04-25T19:06:16.022 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_basic": { 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_basic", 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.023 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.024 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_crash": { 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_crash", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.025 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_device": { 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_device", 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.026 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.027 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_ident": { 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_ident", 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.028 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.029 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_perf": { 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_perf", 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.030 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "contact": { 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "contact", 2024-04-25T19:06:16.031 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.032 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "description": { 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "description", 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.033 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.034 INFO:teuthology.orchestra.run.smithi063.stdout: "device_url": { 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_url", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.035 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.036 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.037 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "24", 2024-04-25T19:06:16.038 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "8", 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "last_opt_revision": { 2024-04-25T19:06:16.039 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "last_opt_revision", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.040 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard": { 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard", 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.041 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.042 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard_description": { 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard_description", 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.043 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.044 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.045 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.046 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.047 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.048 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.049 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.050 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.051 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.052 INFO:teuthology.orchestra.run.smithi063.stdout: "organization": { 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "organization", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.053 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "proxy": { 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "proxy", 2024-04-25T19:06:16.054 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.055 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "url": { 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url", 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.056 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.057 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "test_orchestrator", 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.058 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.059 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.061 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.062 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.063 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.064 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.065 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.066 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "volumes", 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.069 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.070 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.071 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.072 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.073 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.074 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.075 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.076 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.077 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_clones": { 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_clones", 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4", 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.078 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "periodic_async_work": { 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "periodic_async_work", 2024-04-25T19:06:16.079 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Periodically check for async work", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.080 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_delay": { 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.081 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.083 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.084 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix", 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "discovery_interval": { 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "discovery_interval", 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.085 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "100", 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "identifier": { 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "identifier", 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.087 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.088 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.089 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.090 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.091 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.092 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.093 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.094 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.095 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.096 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.097 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_host": { 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_host", 2024-04-25T19:06:16.098 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.099 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_port": { 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_port", 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.100 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10051", 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_sender": { 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_sender", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.102 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.103 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "gid": 25066, 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "z", 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr_features": 4540701547738038271, 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "available_modules": [ 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "alerts", 2024-04-25T19:06:16.104 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.105 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.107 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.108 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.109 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.110 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.112 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.113 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.114 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_destination": { 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_destination", 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.115 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.116 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_from_name": { 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_from_name", 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "Ceph", 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.117 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email From: name", 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_host": { 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_host", 2024-04-25T19:06:16.118 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP server", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.119 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_password": { 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_password", 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.120 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Password to authenticate with", 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.121 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_port": { 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_port", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "465", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.122 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP port", 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_sender": { 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_sender", 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.123 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.124 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_ssl": { 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_ssl", 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.125 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_user": { 2024-04-25T19:06:16.126 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_user", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.127 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "User to authenticate as", 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "balancer", 2024-04-25T19:06:16.128 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "active": { 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "active", 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.129 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.130 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_time": { 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_time", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0000", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.131 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_weekday": { 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_weekday", 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.132 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_max_iterations": { 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.134 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "25", 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "250", 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_metrics": { 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_metrics", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.136 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_step": { 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_step", 2024-04-25T19:06:16.137 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.5", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0.001", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "0.999", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:06:16.138 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "end_time": { 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_time", 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.139 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2359", 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.140 INFO:teuthology.orchestra.run.smithi063.stdout: "end_weekday": { 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_weekday", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.141 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.142 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.143 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.145 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.148 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.149 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.150 INFO:teuthology.orchestra.run.smithi063.stdout: "min_score": { 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_score", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.151 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.152 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "upmap", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-compat", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "none", 2024-04-25T19:06:16.153 INFO:teuthology.orchestra.run.smithi063.stdout: "read", 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap", 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap-read" 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Balancer mode", 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.154 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_ids": { 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_ids", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.155 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.156 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.157 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_deviation": { 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_deviation", 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.158 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.159 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_optimizations": { 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.160 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.161 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm", 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_down_multiplier": { 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_down_multiplier", 2024-04-25T19:06:16.162 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.163 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_refresh_rate": { 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_refresh_rate", 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.164 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "20", 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_starting_port": { 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_starting_port", 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4721", 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.166 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_ptrace": { 2024-04-25T19:06:16.167 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_ptrace", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.168 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_interval": { 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_interval", 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.169 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.170 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.7", 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.171 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm_log_destination": { 2024-04-25T19:06:16.172 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm_log_destination", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "file", 2024-04-25T19:06:16.173 INFO:teuthology.orchestra.run.smithi063.stdout: "file,syslog", 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "syslog" 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.174 INFO:teuthology.orchestra.run.smithi063.stdout: "cgroups_split": { 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cgroups_split", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.175 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "config_checks_enabled": { 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_checks_enabled", 2024-04-25T19:06:16.176 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.177 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "config_dashboard": { 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_dashboard", 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.178 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.179 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_alertmanager": { 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_alertmanager", 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.180 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_base": { 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_base", 2024-04-25T19:06:16.181 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:06:16.182 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_elasticsearch": { 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.183 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "elasticsearch container image", 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_grafana": { 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_grafana", 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.185 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_haproxy": { 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_haproxy", 2024-04-25T19:06:16.186 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "HAproxy container image", 2024-04-25T19:06:16.187 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_agent": { 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.188 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_collector": { 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.190 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_query": { 2024-04-25T19:06:16.191 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.192 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger query container image", 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_keepalived": { 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_keepalived", 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Keepalived container image", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.194 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_loki": { 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_loki", 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.195 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Loki container image", 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.196 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_node_exporter": { 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_node_exporter", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.197 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_nvmeof": { 2024-04-25T19:06:16.198 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_nvmeof", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Nvme-of container image", 2024-04-25T19:06:16.199 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_prometheus": { 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_prometheus", 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.200 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.201 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_promtail": { 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_promtail", 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:06:16.202 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Promtail container image", 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.203 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_samba": { 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_samba", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.204 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_snmp_gateway": { 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:06:16.205 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:06:16.206 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "container_init": { 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_init", 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.207 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "daemon_cache_timeout": { 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.209 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:06:16.210 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "900", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:06:16.211 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "default_registry": { 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_registry", 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.212 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io", 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "device_cache_timeout": { 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_cache_timeout", 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1800", 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.214 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "device_enhanced_scan": { 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_enhanced_scan", 2024-04-25T19:06:16.215 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:06:16.216 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "facts_cache_timeout": { 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "facts_cache_timeout", 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.217 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.218 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "grafana_dashboards_path": { 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.219 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "host_check_interval": { 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "host_check_interval", 2024-04-25T19:06:16.220 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:06:16.221 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "hw_monitoring": { 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hw_monitoring", 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.223 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "inventory_list_all": { 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "inventory_list_all", 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.224 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.225 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.226 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.227 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "log_refresh_metadata": { 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_refresh_metadata", 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.228 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.230 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.231 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.232 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.233 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.234 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.235 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:06:16.237 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "*", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:06:16.238 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "max_count_per_host": { 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_count_per_host", 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.239 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd_draining_count": { 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_osd_draining_count", 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.241 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "migration_current": { 2024-04-25T19:06:16.242 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "migration_current", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.243 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "internal - do not modify", 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.244 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "root", 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm-package", 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "root" 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:06:16.245 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "oob_default_addr": { 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "oob_default_addr", 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.246 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "169.254.1.1", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.247 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "prometheus_alerts_path": { 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:06:16.248 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.249 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_insecure": { 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_insecure", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.250 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_password": { 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_password", 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.251 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.252 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_url": { 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_url", 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.253 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.254 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_username": { 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_username", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.255 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "secure_monitoring_stack": { 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:06:16.256 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.257 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "service_discovery_port": { 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "service_discovery_port", 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.258 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8765", 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.259 INFO:teuthology.orchestra.run.smithi063.stdout: "ssh_config_file": { 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssh_config_file", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.260 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "use_agent": { 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_agent", 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.261 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.262 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "use_repo_digest": { 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_repo_digest", 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.263 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.265 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.265 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:06:16.265 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.265 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.265 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_failed_host_check": { 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.266 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_daemons": { 2024-04-25T19:06:16.267 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.268 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_hosts": { 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.269 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.270 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crash", 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.271 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.272 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.273 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.274 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.275 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.276 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.277 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.278 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.279 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "retain_interval": { 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retain_interval", 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "31536000", 2024-04-25T19:06:16.280 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_recent_interval": { 2024-04-25T19:06:16.281 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_recent_interval", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1209600", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.282 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dashboard", 2024-04-25T19:06:16.283 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.284 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.285 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.286 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:06:16.287 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.288 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.289 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.290 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.291 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:06:16.292 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.293 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:06:16.294 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.295 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.296 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.298 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:06:16.299 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.300 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.301 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.302 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.303 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.304 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.305 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:06:16.306 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.307 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.308 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.309 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.310 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.311 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_URL": { 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.312 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.313 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.313 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.313 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.313 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.388+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 msgr2=0x7f80740a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.388+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 0x7f80740a61e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f80 tx=0x7f807c005fd0 comp rx=0 tx=0).stop 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.388+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 msgr2=0x7f80941a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.388+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f808400e9c0 tx=0x7f808400ee80 comp rx=0 tx=0).stop 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 shutdown_connections 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f80740a3d20 0x7f80740a61e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:16.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8094107100 0x7f809419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:16.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f8094106c90 0x7f80941a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:16.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 --2- 172.21.15.63:0/2395324050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80941047e0 0x7f80941a4790 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:16.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 >> 172.21.15.63:0/2395324050 conn(0x7f8094076790 msgr2=0x7f8094101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:16.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 shutdown_connections 2024-04-25T19:06:16.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:15.389+0000 7f8093200640 1 -- 172.21.15.63:0/2395324050 wait complete. 2024-04-25T19:06:16.318 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:15 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/3882722610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:06:16.318 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:15 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2395324050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:06:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:15 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/3882722610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:06:16.318 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:15 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2395324050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:06:16.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.320 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.321 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.322 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.323 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.324 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.325 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.326 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:06:16.327 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.328 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.329 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "[]", 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.330 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "{}", 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.331 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:06:16.332 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.333 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.334 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.335 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.336 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:06:16.337 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.338 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.339 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.340 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.341 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.343 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:06:16.344 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.345 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.346 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.347 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.348 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.349 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.350 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:06:16.351 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.352 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.353 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8", 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.354 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "45", 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.355 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:06:16.356 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.357 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.358 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.359 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.360 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:06:16.361 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.362 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.363 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.364 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.365 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.366 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.367 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:06:16.368 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.369 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "cross_origin_url": { 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cross_origin_url", 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.370 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.371 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "crt_file": { 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crt_file", 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.372 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "debug": { 2024-04-25T19:06:16.373 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "debug", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.374 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "jwt_token_ttl": { 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "jwt_token_ttl", 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.375 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "28800", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.376 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.377 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.378 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.379 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.380 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.381 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.382 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.383 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.384 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.385 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.386 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "motd": { 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "motd", 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.387 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "The message of the day", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.388 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.389 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.391 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.392 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8080", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.393 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.394 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.395 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl_server_port": { 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl_server_port", 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.396 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8443", 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.397 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "redirect", 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.398 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect" 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.399 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:16.400 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "url_prefix": { 2024-04-25T19:06:16.401 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url_prefix", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.402 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.403 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "devicehealth", 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_monitoring": { 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_monitoring", 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.404 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "monitor device health metrics", 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.405 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.406 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.407 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.408 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.409 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.410 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.411 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.412 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.413 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "mark_out_threshold": { 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mark_out_threshold", 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.414 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2419200", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.415 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": { 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_name", 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "device_health_metrics", 2024-04-25T19:06:16.416 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.417 INFO:teuthology.orchestra.run.smithi063.stdout: "retention_period": { 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retention_period", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15552000", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.418 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_frequency": { 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_frequency", 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.419 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.420 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "self_heal": { 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "self_heal", 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.421 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.422 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.423 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_threshold": { 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_threshold", 2024-04-25T19:06:16.424 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7257600", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:06:16.425 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "diskprediction_local", 2024-04-25T19:06:16.426 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.427 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.428 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.429 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.430 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.431 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.432 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.433 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.434 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "predict_interval": { 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predict_interval", 2024-04-25T19:06:16.435 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.436 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "predictor_model": { 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predictor_model", 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.437 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "prophetstor", 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.439 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.440 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "influx", 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": false, 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "batch_size": { 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "batch_size", 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.441 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5000", 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.442 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "database": { 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "database", 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "ceph", 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.443 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "hostname": { 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hostname", 2024-04-25T19:06:16.444 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.445 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "30", 2024-04-25T19:06:16.446 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "5", 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.447 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.448 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.449 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.451 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.452 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.453 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.454 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "password": { 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "password", 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.456 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "port": { 2024-04-25T19:06:16.457 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "port", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8086", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.458 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server port", 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.459 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "false", 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "threads": { 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threads", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "32", 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.461 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "username": { 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "username", 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.463 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "verify_ssl": { 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "verify_ssl", 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "true", 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.464 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.465 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "insights", 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.466 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.467 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.468 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.469 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.470 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.471 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.472 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.473 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "iostat", 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.474 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.475 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.476 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.477 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.478 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.479 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.480 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.481 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.482 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "k8sevents", 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.483 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_event_retention_days": { 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.484 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "config_check_secs": { 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_check_secs", 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.485 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "10", 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.486 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.487 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.488 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.489 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.490 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.491 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.492 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.493 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.494 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "localpool", 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.495 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "failure_domain": { 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "failure_domain", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "host", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.496 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.498 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.499 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.500 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.501 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.502 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.503 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.504 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.505 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": { 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_size", 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.506 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.507 INFO:teuthology.orchestra.run.smithi063.stdout: "num_rep": { 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "num_rep", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.508 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": { 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_num", 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.509 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "128", 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:06:16.510 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "prefix": { 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prefix", 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.511 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.512 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "subtree": { 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "subtree", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "rack", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.513 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.514 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mds_autoscaler", 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.515 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.516 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.518 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.520 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.521 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.522 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.523 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirroring", 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.524 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.525 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.526 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.527 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.528 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.529 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.530 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.531 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.532 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "nfs", 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.533 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.534 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.535 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.536 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.537 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.539 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.540 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.541 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.542 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "fail_fs": { 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "fail_fs", 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.543 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.544 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.545 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.546 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.547 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.548 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.549 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.550 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.551 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.552 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator": { 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.553 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "rook", 2024-04-25T19:06:16.554 INFO:teuthology.orchestra.run.smithi063.stdout: "test_orchestrator" 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Orchestrator backend", 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_perf_query", 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.556 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.557 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.558 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.559 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.560 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.561 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.562 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.563 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_support", 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.565 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.566 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.567 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.568 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.569 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.570 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.571 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.572 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.573 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_autoscaler", 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.574 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.575 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.576 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.577 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.578 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.579 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.580 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.581 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.582 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.583 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.584 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "threshold": { 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threshold", 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1.0", 2024-04-25T19:06:16.585 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "scaling threshold", 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "progress", 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_pg_recovery_event": { 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.587 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.588 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.589 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.591 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.592 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.593 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.594 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.595 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.596 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.597 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.598 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "max_completed_events": { 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_completed_events", 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.599 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "50", 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.601 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.602 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus", 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "cache": { 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cache", 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.603 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.604 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "exclude_perf_counters": { 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "exclude_perf_counters", 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.606 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.607 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.608 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.609 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.611 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.612 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.613 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.614 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools": { 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools", 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.615 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.616 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "300", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.617 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_interval": { 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_interval", 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.618 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15.0", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.619 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:16.620 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:16.621 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "9283", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.622 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "stale_cache_strategy": { 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stale_cache_strategy", 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.623 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "log", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.624 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "default", 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.625 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "default", 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "error" 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.626 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:16.627 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_support", 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.629 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.630 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.632 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.634 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.635 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.636 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.637 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_snap_create": { 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.638 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:06:16.639 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.640 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "trash_purge_schedule": { 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "trash_purge_schedule", 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.641 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.642 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "restful", 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_auth": { 2024-04-25T19:06:16.643 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_auth", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.644 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.645 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.646 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.647 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.648 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.649 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.650 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.651 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.652 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.653 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.655 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:16.656 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.657 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rgw", 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.658 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.659 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.660 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.661 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.662 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.663 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.664 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.665 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.666 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rook", 2024-04-25T19:06:16.667 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.668 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.669 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.670 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.671 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.672 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.673 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.674 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.675 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_class": { 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "storage_class", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "local", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.676 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.677 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "selftest", 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.678 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.679 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.680 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.681 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.682 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.683 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.684 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.685 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "roption1": { 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption1", 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.686 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.687 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "roption2": { 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption2", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "xyz", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.688 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption1": { 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption1", 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.689 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.690 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption2": { 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption2", 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.691 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption3": { 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption3", 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.692 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.693 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption4": { 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption4", 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.694 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption5": { 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption5", 2024-04-25T19:06:16.695 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.696 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption6": { 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption6", 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.697 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.698 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption7": { 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption7", 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.699 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "42", 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "testkey": { 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testkey", 2024-04-25T19:06:16.700 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.701 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "testlkey": { 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testlkey", 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.702 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.703 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "testnewline": { 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testnewline", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.704 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.705 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snap_schedule", 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_m_granularity": { 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_m_granularity", 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.706 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.707 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "dump_on_update": { 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dump_on_update", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.708 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.709 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.710 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.711 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.712 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.713 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.714 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.716 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stats", 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.717 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.718 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.719 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.720 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.721 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.722 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.723 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.724 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "status", 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.726 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.727 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.728 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.729 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.730 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.731 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.732 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.733 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.734 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telegraf", 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "address": { 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "address", 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.735 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.736 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15", 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.737 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.738 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.739 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.740 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.741 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.742 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.743 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.744 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.745 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telemetry", 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_basic": { 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_basic", 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.747 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.748 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_crash": { 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_crash", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.749 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_device": { 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_device", 2024-04-25T19:06:16.750 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.751 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_ident": { 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_ident", 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.752 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_perf": { 2024-04-25T19:06:16.753 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_perf", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.754 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "contact": { 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "contact", 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.755 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.756 INFO:teuthology.orchestra.run.smithi063.stdout: "description": { 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "description", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.757 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "device_url": { 2024-04-25T19:06:16.758 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_url", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.759 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.760 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.761 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "24", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "8", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.762 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "last_opt_revision": { 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "last_opt_revision", 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.763 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.764 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard": { 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard", 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.765 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard_description": { 2024-04-25T19:06:16.766 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard_description", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.767 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.768 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.769 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.770 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.771 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.772 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.773 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.774 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.775 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "organization": { 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "organization", 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.776 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "proxy": { 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "proxy", 2024-04-25T19:06:16.777 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.778 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "url": { 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url", 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.779 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.780 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "test_orchestrator", 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.781 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.783 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.784 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.785 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.786 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.787 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.788 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.789 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "volumes", 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.790 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.791 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.792 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.793 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.794 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.795 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.797 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.798 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_clones": { 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_clones", 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.799 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4", 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.800 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "periodic_async_work": { 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "periodic_async_work", 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.801 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Periodically check for async work", 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_delay": { 2024-04-25T19:06:16.802 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.803 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.804 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.805 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix", 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.806 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "discovery_interval": { 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "discovery_interval", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "100", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.807 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "identifier": { 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "identifier", 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.808 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.809 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.810 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.811 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.812 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.813 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.814 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.815 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.816 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.817 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.818 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.819 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_host": { 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_host", 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.820 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.821 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_port": { 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_port", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10051", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.822 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_sender": { 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_sender", 2024-04-25T19:06:16.823 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.824 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.825 INFO:teuthology.orchestra.run.smithi063.stdout: "modules": [ 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm", 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "dashboard", 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "iostat", 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "nfs", 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "restful", 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "snap_schedule" 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.826 INFO:teuthology.orchestra.run.smithi063.stdout: "available_modules": [ 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "alerts", 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.827 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.828 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.829 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.830 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.831 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.832 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.833 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.834 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.835 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.836 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_destination": { 2024-04-25T19:06:16.837 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_destination", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.838 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email address to send alerts to", 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_from_name": { 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_from_name", 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.839 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "Ceph", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Email From: name", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.840 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_host": { 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_host", 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP server", 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_password": { 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_password", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Password to authenticate with", 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_port": { 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_port", 2024-04-25T19:06:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "465", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP port", 2024-04-25T19:06:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_sender": { 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_sender", 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SMTP envelope sender", 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_ssl": { 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_ssl", 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.848 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "smtp_user": { 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "smtp_user", 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.849 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "User to authenticate as", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.850 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "balancer", 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.851 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "active": { 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "active", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.852 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_time": { 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_time", 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.853 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0000", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:16.854 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "begin_weekday": { 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "begin_weekday", 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.855 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_max_iterations": { 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_max_iterations", 2024-04-25T19:06:16.856 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "25", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "250", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.857 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_metrics": { 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_metrics", 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.858 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.859 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_compat_step": { 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crush_compat_step", 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.5", 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0.001", 2024-04-25T19:06:16.860 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "0.999", 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "aggressiveness of optimization", 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "end_time": { 2024-04-25T19:06:16.861 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_time", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2359", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.862 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "end_weekday": { 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "end_weekday", 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.863 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "0", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "6", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T19:06:16.864 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.865 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.867 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.868 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.869 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.870 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.871 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.872 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "min_score": { 2024-04-25T19:06:16.873 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_score", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.874 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.875 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "upmap", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "crush-compat", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "none", 2024-04-25T19:06:16.876 INFO:teuthology.orchestra.run.smithi063.stdout: "read", 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap", 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap-read" 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Balancer mode", 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.877 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_ids": { 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_ids", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.878 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.879 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.880 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_deviation": { 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_deviation", 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.882 INFO:teuthology.orchestra.run.smithi063.stdout: "upmap_max_optimizations": { 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "upmap_max_optimizations", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.883 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm", 2024-04-25T19:06:16.884 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_down_multiplier": { 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_down_multiplier", 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.885 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_refresh_rate": { 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_refresh_rate", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "20", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.887 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "agent_starting_port": { 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "agent_starting_port", 2024-04-25T19:06:16.888 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4721", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.889 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_ptrace": { 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_ptrace", 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.890 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.891 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_interval": { 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_interval", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.892 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "autotune_memory_target_ratio": { 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:16.893 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0.7", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.894 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm_log_destination": { 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cephadm_log_destination", 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.895 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "file", 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "file,syslog", 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "syslog" 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.896 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "cgroups_split": { 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cgroups_split", 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.897 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.898 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "config_checks_enabled": { 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_checks_enabled", 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.899 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "config_dashboard": { 2024-04-25T19:06:16.900 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_dashboard", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.901 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_alertmanager": { 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_alertmanager", 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.902 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.903 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_base": { 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_base", 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.904 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Container image name, without the tag", 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_elasticsearch": { 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_elasticsearch", 2024-04-25T19:06:16.905 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "elasticsearch container image", 2024-04-25T19:06:16.906 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_grafana": { 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_grafana", 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.907 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.908 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_haproxy": { 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_haproxy", 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.909 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "HAproxy container image", 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_agent": { 2024-04-25T19:06:16.910 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_agent", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.911 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger agent container image", 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_collector": { 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_collector", 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.912 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger collector container image", 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.913 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_jaeger_query": { 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_jaeger_query", 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T19:06:16.914 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Jaeger query container image", 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.915 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_keepalived": { 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_keepalived", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.916 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Keepalived container image", 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_loki": { 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_loki", 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.917 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Loki container image", 2024-04-25T19:06:16.918 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_node_exporter": { 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_node_exporter", 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.919 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.920 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_nvmeof": { 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_nvmeof", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.921 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Nvme-of container image", 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_prometheus": { 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_prometheus", 2024-04-25T19:06:16.922 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Prometheus container image", 2024-04-25T19:06:16.923 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_promtail": { 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_promtail", 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.924 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Promtail container image", 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.925 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_samba": { 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_samba", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.926 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Samba/SMB container image", 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "container_image_snmp_gateway": { 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_image_snmp_gateway", 2024-04-25T19:06:16.927 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "SNMP Gateway container image", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.928 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "container_init": { 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "container_init", 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.929 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "daemon_cache_timeout": { 2024-04-25T19:06:16.930 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "daemon_cache_timeout", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.931 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "default_cephadm_command_timeout": { 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.932 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "900", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.933 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "default_registry": { 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "default_registry", 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "docker.io", 2024-04-25T19:06:16.934 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.935 INFO:teuthology.orchestra.run.smithi063.stdout: "device_cache_timeout": { 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_cache_timeout", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1800", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.936 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache device inventory", 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "device_enhanced_scan": { 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_enhanced_scan", 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.937 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.938 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "facts_cache_timeout": { 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "facts_cache_timeout", 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:16.939 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "seconds to cache host facts data", 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.940 INFO:teuthology.orchestra.run.smithi063.stdout: "grafana_dashboards_path": { 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "grafana_dashboards_path", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.941 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "host_check_interval": { 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "host_check_interval", 2024-04-25T19:06:16.942 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to perform a host check", 2024-04-25T19:06:16.943 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "hw_monitoring": { 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hw_monitoring", 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.944 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.945 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "inventory_list_all": { 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "inventory_list_all", 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.946 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.947 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.948 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.949 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "log_refresh_metadata": { 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_refresh_metadata", 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.950 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.951 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.952 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.953 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.954 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.955 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.956 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T19:06:16.957 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.958 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.959 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "*", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.960 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "max_count_per_host": { 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_count_per_host", 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.961 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of daemons per service per host", 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "max_osd_draining_count": { 2024-04-25T19:06:16.962 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_osd_draining_count", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T19:06:16.963 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "migration_current": { 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "migration_current", 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.964 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "internal - do not modify", 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "mode": { 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mode", 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "root", 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.966 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm-package", 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "root" 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.967 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "oob_default_addr": { 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "oob_default_addr", 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "169.254.1.1", 2024-04-25T19:06:16.968 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "prometheus_alerts_path": { 2024-04-25T19:06:16.969 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus_alerts_path", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.970 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_insecure": { 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_insecure", 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.971 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.972 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_password": { 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_password", 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.973 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.974 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_url": { 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_url", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.975 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "registry_username": { 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "registry_username", 2024-04-25T19:06:16.976 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.977 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "secure_monitoring_stack": { 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "secure_monitoring_stack", 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.978 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.979 INFO:teuthology.orchestra.run.smithi063.stdout: "service_discovery_port": { 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "service_discovery_port", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8765", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.980 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "cephadm service discovery port", 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "ssh_config_file": { 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssh_config_file", 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.981 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.982 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "use_agent": { 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_agent", 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.983 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.984 INFO:teuthology.orchestra.run.smithi063.stdout: "use_repo_digest": { 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "use_repo_digest", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.985 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_failed_host_check": { 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_failed_host_check", 2024-04-25T19:06:16.986 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.987 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_daemons": { 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_daemons", 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.988 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_on_stray_hosts": { 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_on_stray_hosts", 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.990 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.991 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crash", 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.992 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.993 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.994 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.995 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:16.996 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:16.997 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:16.998 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:16.999 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.000 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "retain_interval": { 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retain_interval", 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "31536000", 2024-04-25T19:06:17.001 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.002 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_recent_interval": { 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_recent_interval", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1209600", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.003 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.004 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dashboard", 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.005 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.006 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.007 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.008 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.009 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_ENABLED": { 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.010 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.011 INFO:teuthology.orchestra.run.smithi063.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.012 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.013 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.015 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T19:06:17.016 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.017 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.018 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.019 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.020 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T19:06:17.021 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.022 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.023 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.024 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.025 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.026 INFO:teuthology.orchestra.run.smithi063.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.027 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T19:06:17.028 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.029 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.030 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.031 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_URL": { 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_URL", 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.032 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T19:06:17.033 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.034 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.035 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.036 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.037 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.038 INFO:teuthology.orchestra.run.smithi063.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.039 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.040 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.041 INFO:teuthology.orchestra.run.smithi063.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "[]", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.042 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.043 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "{}", 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.045 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T19:06:17.046 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.047 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.048 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T19:06:17.049 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.050 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.051 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T19:06:17.052 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.053 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.054 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.055 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.056 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T19:06:17.057 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.058 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.059 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.061 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T19:06:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.064 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.065 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "45", 2024-04-25T19:06:17.066 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.067 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.068 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T19:06:17.069 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "admin", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.070 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.071 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.072 INFO:teuthology.orchestra.run.smithi063.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.073 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.074 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.075 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.076 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T19:06:17.077 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.078 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.079 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.080 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "cross_origin_url": { 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cross_origin_url", 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.081 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "crt_file": { 2024-04-25T19:06:17.082 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "crt_file", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.083 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "debug": { 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "debug", 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.084 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Enable/disable debug options", 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.085 INFO:teuthology.orchestra.run.smithi063.stdout: "jwt_token_ttl": { 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "jwt_token_ttl", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "28800", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.086 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.087 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.088 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.089 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.090 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.091 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.092 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.093 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.094 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.095 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.096 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "motd": { 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "motd", 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.097 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "The message of the day", 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.098 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect_resolve_ip_addr": { 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.099 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.100 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.101 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.102 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8080", 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.104 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl_server_port": { 2024-04-25T19:06:17.105 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl_server_port", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8443", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.106 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.107 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "redirect", 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: "redirect" 2024-04-25T19:06:17.108 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.109 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.110 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "url_prefix": { 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url_prefix", 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.111 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.112 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "devicehealth", 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_monitoring": { 2024-04-25T19:06:17.113 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_monitoring", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.114 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "monitor device health metrics", 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.115 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.116 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.117 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.118 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.119 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.120 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.121 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.122 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.123 INFO:teuthology.orchestra.run.smithi063.stdout: "mark_out_threshold": { 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mark_out_threshold", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "2419200", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.124 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "pool_name": { 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pool_name", 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.125 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "device_health_metrics", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.126 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "retention_period": { 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "retention_period", 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.127 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15552000", 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long to retain device health metrics", 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_frequency": { 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_frequency", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.129 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "self_heal": { 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "self_heal", 2024-04-25T19:06:17.130 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T19:06:17.131 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.132 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.133 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "warn_threshold": { 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "warn_threshold", 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7257600", 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.134 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "diskprediction_local", 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.136 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.137 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.139 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.140 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.141 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.142 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.143 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "predict_interval": { 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predict_interval", 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.144 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "86400", 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.145 INFO:teuthology.orchestra.run.smithi063.stdout: "predictor_model": { 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "predictor_model", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "prophetstor", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.146 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.147 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "600", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.148 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "influx", 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": false, 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "influxdb python module not found", 2024-04-25T19:06:17.149 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "batch_size": { 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "batch_size", 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5000", 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.150 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "database": { 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "database", 2024-04-25T19:06:17.151 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "ceph", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.152 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "hostname": { 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "hostname", 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.153 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server hostname", 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.154 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "30", 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "5", 2024-04-25T19:06:17.155 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.156 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.157 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.158 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.159 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.160 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.161 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.162 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.163 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.164 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "password": { 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "password", 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.165 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "password of InfluxDB server user", 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "port": { 2024-04-25T19:06:17.166 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "port", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "8086", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.167 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "InfluxDB server port", 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "ssl": { 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ssl", 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.168 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "false", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.169 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "threads": { 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threads", 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:17.170 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "32", 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "username": { 2024-04-25T19:06:17.171 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "username", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.172 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "username of InfluxDB server user", 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "verify_ssl": { 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "verify_ssl", 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.173 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "true", 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.174 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "insights", 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.175 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.176 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.177 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.178 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.179 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.180 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.181 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.182 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.183 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.184 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "iostat", 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.185 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.186 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.187 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.188 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.189 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.190 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.191 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.192 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.193 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "k8sevents", 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_event_retention_days": { 2024-04-25T19:06:17.194 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "ceph_event_retention_days", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "7", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T19:06:17.195 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "config_check_secs": { 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "config_check_secs", 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.196 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "10", 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.197 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.198 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.199 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.200 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.201 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.202 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.203 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.204 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.205 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.206 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "localpool", 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "failure_domain": { 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "failure_domain", 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.207 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "host", 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "failure domain for any created local pool", 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.209 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.210 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.211 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.212 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.213 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.214 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.215 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.216 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "min_size": { 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "min_size", 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.217 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default min_size for any created local pool", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T19:06:17.218 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "num_rep": { 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "num_rep", 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3", 2024-04-25T19:06:17.219 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default replica count for any created local pool", 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_num": { 2024-04-25T19:06:17.220 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_num", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "128", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.221 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "prefix": { 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prefix", 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.222 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "name prefix for any created local pool", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.223 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "subtree": { 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "subtree", 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "rack", 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.224 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.225 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mds_autoscaler", 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.226 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.227 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.228 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.229 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.230 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.231 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.232 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.233 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.234 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirroring", 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.235 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.236 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.237 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.238 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.239 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.240 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.241 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.242 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.243 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "nfs", 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.244 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.245 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.246 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.247 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.248 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.249 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.250 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.251 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.252 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.253 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "fail_fs": { 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "fail_fs", 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.254 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.255 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.256 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.257 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.258 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.260 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.261 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.262 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.263 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator": { 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "orchestrator", 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.264 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "cephadm", 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "rook", 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "test_orchestrator" 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Orchestrator backend", 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.265 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_perf_query", 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.266 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.267 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.268 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.269 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.270 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.271 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.272 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.274 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "osd_support", 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.276 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.277 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.278 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.279 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.280 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.281 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.282 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.283 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.284 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "pg_autoscaler", 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.285 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.286 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.287 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.288 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.289 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.290 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.291 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.292 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "threshold": { 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "threshold", 2024-04-25T19:06:17.295 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "3.0", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1.0", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "scaling threshold", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T19:06:17.296 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "progress", 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.297 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_pg_recovery_event": { 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_pg_recovery_event", 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.298 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:17.299 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.300 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.301 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.302 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.303 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.304 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.305 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.306 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.307 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.308 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "max_completed_events": { 2024-04-25T19:06:17.309 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_completed_events", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "50", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.310 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "number of past completed events to remember", 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "sleep_interval": { 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "sleep_interval", 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.311 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "5", 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "how long the module is going to sleep", 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.312 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "prometheus", 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.313 INFO:teuthology.orchestra.run.smithi063.stdout: "cache": { 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "cache", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.314 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "exclude_perf_counters": { 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "exclude_perf_counters", 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.315 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.316 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.317 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.318 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.319 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.320 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.321 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.322 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.323 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.324 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools": { 2024-04-25T19:06:17.325 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.326 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.327 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "300", 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.328 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "scrape_interval": { 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "scrape_interval", 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15.0", 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.329 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:17.330 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "::", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.331 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.332 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "9283", 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.333 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "stale_cache_strategy": { 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stale_cache_strategy", 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "log", 2024-04-25T19:06:17.334 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.335 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_behaviour": { 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_behaviour", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "default", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.336 INFO:teuthology.orchestra.run.smithi063.stdout: "default", 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "error" 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.337 INFO:teuthology.orchestra.run.smithi063.stdout: "standby_error_status_code": { 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "standby_error_status_code", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "500", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "400", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "599", 2024-04-25T19:06:17.338 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.339 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_support", 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.340 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.341 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.342 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.343 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.344 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.345 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.346 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.347 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_snap_create": { 2024-04-25T19:06:17.348 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_snap_create", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.349 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "mirror_snapshot_schedule": { 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.350 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.351 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "trash_purge_schedule": { 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "trash_purge_schedule", 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.352 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "restful", 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "enable_auth": { 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enable_auth", 2024-04-25T19:06:17.354 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.355 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "key_file": { 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "key_file", 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.356 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.358 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.359 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.360 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.361 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.362 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.363 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.364 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.365 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "server_addr": { 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_addr", 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.366 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.367 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: "server_port": { 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "server_port", 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.368 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.369 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rgw", 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.370 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.371 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.372 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.373 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.374 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.375 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.376 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.377 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.378 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.379 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rook", 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.380 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.381 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.382 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.383 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.384 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.385 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.386 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.387 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.388 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "storage_class": { 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "storage_class", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "local", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.389 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.390 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "selftest", 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.391 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.392 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.393 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.394 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.395 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.396 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.397 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.398 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.399 INFO:teuthology.orchestra.run.smithi063.stdout: "roption1": { 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption1", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.400 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "roption2": { 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "roption2", 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.401 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "xyz", 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.402 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption1": { 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption1", 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.403 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.404 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption2": { 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption2", 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.405 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption3": { 2024-04-25T19:06:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption3", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "float", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption4": { 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption4", 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.409 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption5": { 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption5", 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.410 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.411 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption6": { 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption6", 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.412 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "rwoption7": { 2024-04-25T19:06:17.413 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rwoption7", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "1", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "42", 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.414 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "testkey": { 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testkey", 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.415 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.416 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "testlkey": { 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testlkey", 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.417 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.418 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "testnewline": { 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "testnewline", 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.419 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.420 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snap_schedule", 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "allow_m_granularity": { 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "allow_m_granularity", 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.421 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.422 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "dump_on_update": { 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "dump_on_update", 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.423 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "dump database to debug log on update", 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.424 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.425 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.426 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.427 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.428 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.429 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.430 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.431 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.432 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "stats", 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.433 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.434 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.435 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.436 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.437 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.438 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.439 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.440 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.441 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.442 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "status", 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.443 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.444 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.445 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.446 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.447 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.448 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.449 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.450 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.451 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telegraf", 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "address": { 2024-04-25T19:06:17.452 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "address", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.453 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.454 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "15", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.455 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.456 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.457 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.458 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.459 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.460 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.461 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.463 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.464 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "telemetry", 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_basic": { 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_basic", 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.465 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.466 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_crash": { 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_crash", 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.467 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_device": { 2024-04-25T19:06:17.468 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_device", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.469 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_ident": { 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_ident", 2024-04-25T19:06:17.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.471 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "channel_perf": { 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "channel_perf", 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.472 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "contact": { 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "contact", 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.474 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "description": { 2024-04-25T19:06:17.475 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "description", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.476 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "device_url": { 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "device_url", 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.477 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.478 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "enabled": { 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "enabled", 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.479 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:17.480 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "24", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "8", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.481 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "last_opt_revision": { 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "last_opt_revision", 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.482 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "1", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.483 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard": { 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard", 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.484 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "leaderboard_description": { 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "leaderboard_description", 2024-04-25T19:06:17.485 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.486 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.487 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.488 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.489 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.490 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.491 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.492 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.493 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.494 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.495 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.495 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.495 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.495 INFO:teuthology.orchestra.run.smithi063.stdout: "organization": { 2024-04-25T19:06:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "organization", 2024-04-25T19:06:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.497 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "proxy": { 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "proxy", 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.499 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "url": { 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "url", 2024-04-25T19:06:17.500 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.501 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.502 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "test_orchestrator", 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.503 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.505 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.506 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.507 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.508 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.509 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.510 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.511 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "volumes", 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.512 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.513 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.514 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.515 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.516 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.517 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.518 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.519 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.520 INFO:teuthology.orchestra.run.smithi063.stdout: "max_concurrent_clones": { 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "max_concurrent_clones", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "4", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.521 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "periodic_async_work": { 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "periodic_async_work", 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.522 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Periodically check for async work", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.523 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_delay": { 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_delay", 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.524 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "0", 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.525 INFO:teuthology.orchestra.run.smithi063.stdout: "snapshot_clone_no_wait": { 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "True", 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.526 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.527 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix", 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "can_run": true, 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "error_string": "", 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "module_options": { 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "discovery_interval": { 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "discovery_interval", 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "uint", 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.528 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "100", 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.529 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "identifier": { 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "identifier", 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.530 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.531 INFO:teuthology.orchestra.run.smithi063.stdout: "interval": { 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "interval", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "secs", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "60", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.532 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "log_level": { 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_level", 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.533 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.534 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster": { 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster", 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.536 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_cluster_level": { 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_cluster_level", 2024-04-25T19:06:17.537 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "info", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [ 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "", 2024-04-25T19:06:17.538 INFO:teuthology.orchestra.run.smithi063.stdout: "critical", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "debug", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "error", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "info", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "warning" 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.539 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "log_to_file": { 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "log_to_file", 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "bool", 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 1, 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "False", 2024-04-25T19:06:17.540 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_host": { 2024-04-25T19:06:17.541 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_host", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.542 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_port": { 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_port", 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "int", 2024-04-25T19:06:17.543 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "10051", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "zabbix_sender": { 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "zabbix_sender", 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "str", 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "level": "advanced", 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "flags": 0, 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "min": "", 2024-04-25T19:06:17.545 INFO:teuthology.orchestra.run.smithi063.stdout: "max": "", 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: "enum_allowed": [], 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: "desc": "", 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: "long_desc": "", 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: "tags": [], 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: "see_also": [] 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.546 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "services": { 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "dashboard": "https://172.21.15.63:8443/" 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "always_on_modules": { 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus": [ 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "balancer", 2024-04-25T19:06:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "devicehealth", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscaler", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "progress", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_support", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "status", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "telemetry", 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: "volumes" 2024-04-25T19:06:17.548 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific": [ 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "balancer", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "devicehealth", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscaler", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "progress", 2024-04-25T19:06:17.549 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_support", 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "status", 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "telemetry", 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "volumes" 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy": [ 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "balancer", 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-04-25T19:06:17.550 INFO:teuthology.orchestra.run.smithi063.stdout: "devicehealth", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscaler", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "progress", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_support", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "status", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "telemetry", 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "volumes" 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "reef": [ 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "balancer", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "devicehealth", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscaler", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "progress", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_support", 2024-04-25T19:06:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "status", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "telemetry", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "volumes" 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "squid": [ 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "balancer", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "crash", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "devicehealth", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "orchestrator", 2024-04-25T19:06:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "pg_autoscaler", 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "progress", 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "rbd_support", 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "status", 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "telemetry", 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "volumes" 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "last_failure_osd_epoch": 97, 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "active_clients": [ 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "devicehealth", 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 478546311 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "libcephsqlite", 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 4178756473 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "rbd_support", 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2954443945 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snap_schedule", 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 127907254 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "snap_schedule", 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 3234387293 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "volumes", 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:0", 2024-04-25T19:06:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 2568115086 2024-04-25T19:06:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T19:06:17.562 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:17.568 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T19:06:17.569 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T19:06:17.569 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:15 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/3882722610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T19:06:17.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:15 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2395324050' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T19:06:17.570 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:16 smithi063 ceph-mon[28006]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:17.571 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:16 smithi070 ceph-mon[32820]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:17.571 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:16 smithi114 ceph-mon[32988]: pgmap v624: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:18.038 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:18 smithi114 ceph-mon[32988]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:18 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:18 smithi063 ceph-mon[28006]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:18 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:18 smithi070 ceph-mon[32820]: pgmap v625: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 4.4 MiB/s wr, 161 op/s 2024-04-25T19:06:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:18 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:06:19.627 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:19 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:19.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:19 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:19.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:19 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:06:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:19 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:19 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:19 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:06:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:19 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:06:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:19 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:06:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:19 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:06:20.232 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:20.232 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:20.232 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 0, 2024-04-25T19:06:20.232 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2, 2024-04-25T19:06:20.232 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 0, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 0, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.004657336, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.001000720, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:20.233 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:20.234 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 401, 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 388, 2024-04-25T19:06:20.235 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1643072, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 239280, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.157074251, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.062689027, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.093201126, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.018353191, 2024-04-25T19:06:20.236 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 388, 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.038948610, 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000100383 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:20.237 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1643072, 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 239280 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2, 2024-04-25T19:06:20.238 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 295, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 320, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 208000, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.056743360, 2024-04-25T19:06:20.239 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.044179428, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.004748784, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 292, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.053736010, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000184027 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.240 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 208000 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.241 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 0, 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:20.242 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:20.243 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:20.244 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:20.245 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:20.245 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:20.245 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:20.245 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:20.245 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:20.246 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:20.246 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:20.246 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:20.246 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:20.246 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:20.247 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 148731, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 31, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:20.248 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2, 2024-04-25T19:06:20.249 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T19:06:20.250 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 399, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1583996, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 399, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 399, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1583996, 2024-04-25T19:06:20.251 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:20.252 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:20.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:20 smithi063 ceph-mon[28006]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:20 smithi114 ceph-mon[32988]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:20 smithi070 ceph-mon[32820]: pgmap v626: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:21.098 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-25T19:06:21.098 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-25T19:06:21.098 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-25T19:06:21.099 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-25T19:06:21.099 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-25T19:06:21.099 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-25T19:06:21.099 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-25T19:06:21.099 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T19:06:21.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T19:06:21.708 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:22 smithi114 ceph-mon[32988]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:22.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:22 smithi063 ceph-mon[28006]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:22.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:22 smithi070 ceph-mon[32820]: pgmap v627: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:23.930 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:23.930 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:23.930 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 408, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 395, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1644976, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 243760, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.153697657, 2024-04-25T19:06:23.931 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.043144094, 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.114018319, 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.019638126, 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 395, 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.041870915, 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000106002 2024-04-25T19:06:23.932 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1644976, 2024-04-25T19:06:23.933 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 243760 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2, 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 299, 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 320, 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 208704, 2024-04-25T19:06:23.934 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.059151637, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.046320477, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.004299110, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:23.935 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 296, 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.049657530, 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000167761 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:23.936 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 208704 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 0, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 0, 2024-04-25T19:06:23.937 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 0, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.001119436, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.000958938, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:23.938 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:23.939 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 0, 2024-04-25T19:06:23.940 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:23.941 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:23.942 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:23.943 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:23.944 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:23.945 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 148714, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 30, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T19:06:23.946 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:23.947 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:23.948 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 406, 2024-04-25T19:06:23.949 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1584906, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 406, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 406, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1584906, 2024-04-25T19:06:23.950 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:23.951 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:24.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:24 smithi063 ceph-mon[28006]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:24 smithi114 ceph-mon[32988]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:24.931 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-25T19:06:24.932 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-25T19:06:24.933 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T19:06:24.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T19:06:24.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:24 smithi070 ceph-mon[32820]: pgmap v628: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-04-25T19:06:25.422 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:26.612 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:26 smithi063 ceph-mon[28006]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 5.8 MiB/s wr, 186 op/s 2024-04-25T19:06:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:26 smithi114 ceph-mon[32988]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 5.8 MiB/s wr, 186 op/s 2024-04-25T19:06:26.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:26 smithi070 ceph-mon[32820]: pgmap v629: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 5.8 MiB/s wr, 186 op/s 2024-04-25T19:06:27.655 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 0, 2024-04-25T19:06:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2, 2024-04-25T19:06:27.655 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 0, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 0, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.004194460, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.000800293, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:27.656 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:27.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 404, 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 391, 2024-04-25T19:06:27.658 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1643888, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 241200, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.155594793, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.061987630, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.092237931, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.018450258, 2024-04-25T19:06:27.659 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 391, 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.041102321, 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000105121 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:27.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1643888, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 241200 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 3, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 298, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 480, 2024-04-25T19:06:27.661 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 286832, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.055976878, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.043386736, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.004130953, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:27.662 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 295, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.058411920, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000198006 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.663 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 286832 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 0, 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:27.664 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:27.665 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:27.666 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 149745, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 31, 2024-04-25T19:06:27.669 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:27.670 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 0, 2024-04-25T19:06:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:27.671 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:27.672 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 403, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1584413, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 403, 2024-04-25T19:06:27.673 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 403, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1584413, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:27.674 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:27.675 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:27.675 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-04-25T19:06:28.523 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-04-25T19:06:28.524 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-04-25T19:06:28.524 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T19:06:28.524 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T19:06:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:28 smithi114 ceph-mon[32988]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 4.7 MiB/s wr, 145 op/s 2024-04-25T19:06:28.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:28 smithi063 ceph-mon[28006]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 4.7 MiB/s wr, 145 op/s 2024-04-25T19:06:28.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:28 smithi070 ceph-mon[32820]: pgmap v630: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 4.7 MiB/s wr, 145 op/s 2024-04-25T19:06:29.012 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:30.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:30 smithi063 ceph-mon[28006]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 6.7 MiB/s wr, 199 op/s 2024-04-25T19:06:30.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:30 smithi114 ceph-mon[32988]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 6.7 MiB/s wr, 199 op/s 2024-04-25T19:06:30.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:30 smithi070 ceph-mon[32820]: pgmap v631: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 6.7 MiB/s wr, 199 op/s 2024-04-25T19:06:31.259 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T19:06:31.259 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 408, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 397, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1643168, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 245536, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:31.260 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.157729660, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.044952438, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.116776878, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.019668073, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 396, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.039401940, 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000099499 2024-04-25T19:06:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1643168, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 245536 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.262 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 303, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 320, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 209408, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.060283089, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.047797310, 2024-04-25T19:06:31.263 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.003894252, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 300, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.050064343, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000166881 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:31.264 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 209408 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 4, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 4, 2024-04-25T19:06:31.265 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 2992, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 448, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.001229777, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.000051902, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.001120814, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:31.266 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001480349, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000493449 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T19:06:31.267 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 0, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:31.268 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:31.269 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:31.270 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:31.271 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 148714, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 30, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:31.272 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 0, 2024-04-25T19:06:31.273 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2, 2024-04-25T19:06:31.274 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T19:06:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T19:06:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T19:06:31.275 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T19:06:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T19:06:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T19:06:31.276 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 410, 2024-04-25T19:06:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1585529, 2024-04-25T19:06:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 410, 2024-04-25T19:06:31.277 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T19:06:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T19:06:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 410, 2024-04-25T19:06:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1585529, 2024-04-25T19:06:31.278 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T19:06:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T19:06:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T19:06:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T19:06:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:31.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T19:06:31.280 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T19:06:32.224 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T19:06:32.224 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T19:06:32.224 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-25T19:06:32.225 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-25T19:06:32.225 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-25T19:06:32.225 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-25T19:06:32.225 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-25T19:06:32.225 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T19:06:32.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T19:06:32.456 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T19:06:32.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:32 smithi114 ceph-mon[32988]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:32.867 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:32 smithi070 ceph-mon[32820]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:32.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:32 smithi063 ceph-mon[28006]: pgmap v632: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:34.393 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 186416, 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 217207, 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 49369580, 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 302522265, 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 226, 2024-04-25T19:06:34.394 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 10.730283953, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 4.584727243, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 5.969078092, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.934961667, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 217208, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 16.414708387, 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000075571 2024-04-25T19:06:34.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 330849, 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.046427486, 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000140 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 42224, 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 1216 2024-04-25T19:06:34.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 198380, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 221214, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 54621497, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 233559180, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 8, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 6, 2024-04-25T19:06:34.397 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 10.781439811, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 4.309165552, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 6.526925276, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.968558628, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 221216, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 20.380389693, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000092128 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.398 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 351917, 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.046584297, 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000132 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1869760, 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 783520 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:34.399 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 277488, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 385097, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 90365290, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 358858962, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 30, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 8, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 15.311840030, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 6.788893665, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 9.704198725, 2024-04-25T19:06:34.400 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.858359224, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 385089, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 20.303480813, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000052724 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 501170, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.079516069, 2024-04-25T19:06:34.401 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000158 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 730496, 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 411120 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-25T19:06:34.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 81808, 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 90.648478402, 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.001108063 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.403 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 40739, 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 5.278275897, 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000129563 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 187873, 2024-04-25T19:06:34.404 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 164279, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 164279, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 1249.454512966, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.007605686 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 23594, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 24, 2024-04-25T19:06:34.405 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 3173, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 16, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 12, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 2771, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 1172, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 1538, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 61, 2024-04-25T19:06:34.406 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 1097, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 3642, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 238, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 239, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 14, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 286251, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 167023, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 23540, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 126, 2024-04-25T19:06:34.407 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 6, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 2371, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 24903, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 5, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 30, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 2685, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 47, 2024-04-25T19:06:34.408 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 2096, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 666, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 211520610, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 7, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:34.409 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 62, 2024-04-25T19:06:34.410 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 22811, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 82289, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 52587, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 16756, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 8012, 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 107179 2024-04-25T19:06:34.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 273, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 237, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 142, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 268, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 1261, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 77, 2024-04-25T19:06:34.412 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 77, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 22534, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 22457, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:34.413 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 13, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 1353, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 32, 2024-04-25T19:06:34.414 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 28, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 119330, 2024-04-25T19:06:34.415 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 117843, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 117843, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 1487, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 3729, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 238, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 236, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 236, 2024-04-25T19:06:34.416 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 2, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 1, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 5, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 463597563, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 467923922, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 0, 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-04-25T19:06:34.417 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 43639, 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 264.517498897, 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.006061493 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 0 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 1870, 2024-04-25T19:06:34.418 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 26484, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 24614, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 265, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 541, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 276, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 2771, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 54155, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 51384, 2024-04-25T19:06:34.419 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 239, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 25031, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 24792, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 93360, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 231800 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 259641, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 22789, 2024-04-25T19:06:34.420 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 187873, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 2690, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 45648, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 23217, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 383.911932757, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.016535811 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.421 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 20963, 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 183.587546351, 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.008757694 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.422 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-04-25T19:06:34.423 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 44898, 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 164.245839431, 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.003658199 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000419355, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000209677 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.425 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.427 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:34.427 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.427 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 20, 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.016813453, 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000840672 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.428 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 119, 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.446320658, 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.003750593 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.429 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 20, 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 1.921219449, 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.096060972 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-04-25T19:06:34.430 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 11, 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.024252338, 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.002204758 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 47549, 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 190.021944713, 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.003996339 2024-04-25T19:06:34.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 4332, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 100.864025652, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.023283477 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.432 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-04-25T19:06:34.433 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1, 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.001075344, 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.001075344 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 13, 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.390246809, 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.030018985 2024-04-25T19:06:34.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 77, 2024-04-25T19:06:34.435 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 26.484119474, 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.343949603 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.436 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 516, 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 1.377525924, 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.002669623 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.437 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 7, 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.020422292, 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.002917470 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-04-25T19:06:34.438 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 22534, 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 196.140808966, 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.008704216 2024-04-25T19:06:34.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.440 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 2, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 5, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 3, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 2, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 24271, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 12135, 2024-04-25T19:06:34.441 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 1187, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 480, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 480, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:34.442 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:34.443 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:34.444 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:34.445 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:34.446 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 2346998, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 2082, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 176, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 2, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:34.447 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 173, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 7547092, 2024-04-25T19:06:34.448 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 83896, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-04-25T19:06:34.449 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 85284, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 490814363, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 35, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 85284, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 85284, 2024-04-25T19:06:34.450 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 1501.178058429, 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.017602106 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 85284, 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 117655 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.451 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 85284, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 724, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 84560, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 2902, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 12117, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 700, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 44339, 2024-04-25T19:06:34.452 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 762, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 16, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 23278, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:34.453 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 15, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 24234, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-04-25T19:06:34.454 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-04-25T19:06:34.455 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-04-25T19:06:34.456 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 127, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 35, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 12, 2024-04-25T19:06:34.457 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 12, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 0, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 7352, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 48, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 1892 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-04-25T19:06:34.458 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 1, 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 2218 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-04-25T19:06:34.459 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 10, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 22457, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 22457, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:34.460 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 662280, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 134640533, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 662280, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 662280, 2024-04-25T19:06:34.461 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 134640533, 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:34.462 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 85284, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 493084736, 2024-04-25T19:06:34.463 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 85284, 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 493084736, 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:34.464 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 85284, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 85284, 2024-04-25T19:06:34.465 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 85284, 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 85284, 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.466 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 22457, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 2270757, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 22457, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:34.467 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 700, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 2270757, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:34.468 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x5612c3ae6500": { 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 574, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 119330, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 463729618, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 119330, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:34.469 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 43639, 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 463729044, 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:34.470 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-25T19:06:34.692 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:34 smithi070 ceph-mon[32820]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:34.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:34 smithi114 ceph-mon[32988]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:34.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:34 smithi063 ceph-mon[28006]: pgmap v633: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 4.7 MiB/s wr, 132 op/s 2024-04-25T19:06:35.072 INFO:tasks.check_counter:Daemon mds.b mds.exported=30 2024-04-25T19:06:35.072 INFO:tasks.check_counter:Daemon mds.b mds.imported=47 2024-04-25T19:06:35.072 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=273 2024-04-25T19:06:35.072 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=237 2024-04-25T19:06:35.073 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=7 2024-04-25T19:06:35.073 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=20 2024-04-25T19:06:35.073 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=13 2024-04-25T19:06:35.073 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T19:06:35.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T19:06:35.315 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T19:06:36.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:36 smithi070 ceph-mon[32820]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 6.3 MiB/s wr, 199 op/s 2024-04-25T19:06:36.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:36 smithi063 ceph-mon[28006]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 6.3 MiB/s wr, 199 op/s 2024-04-25T19:06:36.911 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-25T19:06:36.911 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1328, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1328, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 154952840, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 421544, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 9, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.177368898, 2024-04-25T19:06:36.912 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.035047948, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.101086757, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.030957084, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1327, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.044132399, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000033257 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.913 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T19:06:36.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1686, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1986, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 143064989, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1220637, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 9, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 7, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.231621739, 2024-04-25T19:06:36.915 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.045322121, 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.123294970, 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.048423427, 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1983, 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.090281333, 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000045527 2024-04-25T19:06:36.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 685120 2024-04-25T19:06:36.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2791, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2740, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 132615123, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 984560, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.351220856, 2024-04-25T19:06:36.918 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.084798280, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.173336066, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.080873762, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 2739, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.107398309, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000039210 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.919 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1849536, 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 251888 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.920 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 8750, 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.214107492, 2024-04-25T19:06:36.921 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000024469 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.922 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.923 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-04-25T19:06:36.924 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 22147, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 1477, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 20670, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 20037, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-04-25T19:06:36.925 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 7, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:36.927 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:36.928 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:36.929 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-04-25T19:06:36.930 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-04-25T19:06:36.931 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-04-25T19:06:36.932 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-04-25T19:06:36.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 111295, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-04-25T19:06:36.934 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 142, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 11, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-04-25T19:06:36.935 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 21709817, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 431400836, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 431400237, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 111295 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 1696, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 22467, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 20771, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 125, 2024-04-25T19:06:36.937 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 248, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 123, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 22147, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 22975, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 828, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-04-25T19:06:36.938 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 86044, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215416 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-04-25T19:06:36.939 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.940 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-04-25T19:06:36.941 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.943 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.944 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.945 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.946 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-04-25T19:06:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.947 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-04-25T19:06:36.948 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.949 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.950 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.951 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.952 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.953 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.954 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-04-25T19:06:36.955 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-04-25T19:06:36.956 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 4, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 3, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 3, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-04-25T19:06:36.959 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 6386, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:36.960 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:36.961 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:36.962 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:36.963 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 314595, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 2034, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-25T19:06:36.964 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 173, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:36.965 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 20033106, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 141103, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-25T19:06:36.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 5312, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 5312, 2024-04-25T19:06:36.967 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5312, 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 6.565807222, 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.001236032 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-04-25T19:06:36.968 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5312, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 5313 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 5312, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 5312, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-04-25T19:06:36.969 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 3120, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 2190, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-04-25T19:06:36.970 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-04-25T19:06:36.971 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:36.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-04-25T19:06:36.973 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-04-25T19:06:36.974 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 127, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 35, 2024-04-25T19:06:36.975 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 11, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 12, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 1, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-04-25T19:06:36.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-04-25T19:06:36.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:36.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 5801, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 430083146, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:36.979 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 5801, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5801, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 430083146, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.980 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:36.981 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5312, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 427310732, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5312, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 427310732, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.982 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:36.983 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5312, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 5312, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5312, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 5312, 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:36.984 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:36.985 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:36.986 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x56327e9666e0": { 2024-04-25T19:06:36.987 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-25T19:06:36.988 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:36.989 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-25T19:06:37.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:36 smithi114 ceph-mon[32988]: pgmap v634: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 6.3 MiB/s wr, 199 op/s 2024-04-25T19:06:37.700 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T19:06:37.700 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T19:06:37.700 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-25T19:06:37.701 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-25T19:06:37.701 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-25T19:06:37.701 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-25T19:06:37.701 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T19:06:37.701 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T19:06:37.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T19:06:37.935 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T19:06:38.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:38 smithi070 ceph-mon[32820]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.7 MiB/s wr, 121 op/s 2024-04-25T19:06:38.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:38 smithi063 ceph-mon[28006]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.7 MiB/s wr, 121 op/s 2024-04-25T19:06:39.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:38 smithi114 ceph-mon[32988]: pgmap v635: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 3.7 MiB/s wr, 121 op/s 2024-04-25T19:06:39.701 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-25T19:06:39.701 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:39.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 3675, 2024-04-25T19:06:39.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 3677, 2024-04-25T19:06:39.701 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 198245333, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 1168657, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 8, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 4, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.380509702, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.088854096, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.179165740, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.087822201, 2024-04-25T19:06:39.702 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 3675, 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.113283635, 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000030825 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.703 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 1378, 2024-04-25T19:06:39.704 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 1327, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 114091697, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 534966, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 6, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.186953158, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.041402408, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.115801652, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.031340128, 2024-04-25T19:06:39.705 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 1326, 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.040567998, 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000030594 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.706 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1849264, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 251248 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 527, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 824, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 79839207, 2024-04-25T19:06:39.707 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 852686, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 9, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 6, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.089225828, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.022339787, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.049482266, 2024-04-25T19:06:39.708 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.011432266, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 822, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.056719638, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000069001 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.709 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 687008 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 1, 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-25T19:06:39.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-MDSRank": { 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 8525, 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.181796195, 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000021325 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "queue_len": 0, 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "complete_latency": { 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.712 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": { 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "request": 0, 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "reply": 0, 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "reply_latency": { 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.713 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_reply": 0, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "forward": 0, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_complete": 0, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_commit": 0, 2024-04-25T19:06:39.714 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_split": 0, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_merge": 0, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 19844, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_top": 793, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_bottom": 19051, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pin_tail": 0, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_pinned": 18712, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_expired": 0, 2024-04-25T19:06:39.715 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_with_caps": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "subtrees": 7, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_hit": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_forward": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_discover": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_dir_fetch": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:39.716 INFO:teuthology.orchestra.run.smithi070.stdout: "traverse_lock": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "load_cent": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "q": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "exported": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "exported_inodes": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "imported": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "imported_inodes": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:39.717 INFO:teuthology.orchestra.run.smithi070.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rfiles": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rbytes": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "root_rsnaps": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:39.718 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_inode_file_caps": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_grant": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:39.719 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps": 0, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_caps_dirty": 0, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_cap_release": 0, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "process_request_cap_release": 0 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_cache": { 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update": 0, 2024-04-25T19:06:39.720 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_update_receipt": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_try_discover": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_send_discover": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "dir_handle_discover": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_delayed": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_created": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_enqueued": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:39.721 INFO:teuthology.orchestra.run.smithi070.stdout: "strays_migrated": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_started": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "recovery_completed": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_path": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_quiesce_inode": 0, 2024-04-25T19:06:39.722 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_exportdir": 0, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_flush": 0, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragmentdir": 0, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "ireq_inodestats": 0 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_log": { 2024-04-25T19:06:39.723 INFO:teuthology.orchestra.run.smithi070.stdout: "evlrg": 0, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "evadd": 0, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "evex": 0, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "evtrm": 0, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "ev": 103540, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "evexg": 0, 2024-04-25T19:06:39.724 INFO:teuthology.orchestra.run.smithi070.stdout: "evexd": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segadd": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segex": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segtrm": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "seg": 130, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segmjr": 10, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segexg": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "segexd": 0, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "expos": 21029039, 2024-04-25T19:06:39.725 INFO:teuthology.orchestra.run.smithi070.stdout: "wrpos": 393020421, 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "rdpos": 393015216, 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "jlat": { 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: "replayed": 103540 2024-04-25T19:06:39.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_mem": { 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "ino": 1451, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "ino+": 20786, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "ino-": 19335, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "dir": 92, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "dir+": 236, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "dir-": 144, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "dn": 19844, 2024-04-25T19:06:39.727 INFO:teuthology.orchestra.run.smithi070.stdout: "dn+": 21299, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "dn-": 1455, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "cap": 0, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "cap+": 0, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "cap-": 0, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "rss": 77052, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "heap": 215416 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_server": { 2024-04-25T19:06:39.728 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_client_request": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "dispatch_server_request": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_request": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_client_session": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "handle_peer_request": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "req_create_latency": { 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.729 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getattr_latency": { 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.730 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "req_link_latency": { 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookup_latency": { 2024-04-25T19:06:39.731 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.732 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupino_latency": { 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupname_latency": { 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.733 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "req_lssnap_latency": { 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.735 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.736 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mkdir_latency": { 2024-04-25T19:06:39.736 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mknod_latency": { 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.737 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: "req_mksnap_latency": { 2024-04-25T19:06:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: "req_open_latency": { 2024-04-25T19:06:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.739 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.740 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.740 INFO:teuthology.orchestra.run.smithi070.stdout: "req_readdir_latency": { 2024-04-25T19:06:39.740 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.740 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.741 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.741 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rename_latency": { 2024-04-25T19:06:39.741 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.741 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.742 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.742 INFO:teuthology.orchestra.run.smithi070.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:39.742 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.742 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.743 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.743 INFO:teuthology.orchestra.run.smithi070.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:39.743 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.743 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmdir_latency": { 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.744 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.745 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:39.745 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.745 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.745 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.746 INFO:teuthology.orchestra.run.smithi070.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:39.746 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.746 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.746 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.747 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setattr_latency": { 2024-04-25T19:06:39.747 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.747 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.747 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.749 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:39.749 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.749 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.749 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.750 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setlayout_latency": { 2024-04-25T19:06:39.750 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.750 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.750 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: "req_setxattr_latency": { 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.751 INFO:teuthology.orchestra.run.smithi070.stdout: "req_symlink_latency": { 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "req_unlink_latency": { 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.753 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_sessions": { 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "session_count": 1, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "session_add": 3, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "session_remove": 2, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_open": 3, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "sessions_stale": 0, 2024-04-25T19:06:39.754 INFO:teuthology.orchestra.run.smithi070.stdout: "total_load": 0, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "average_load": 0, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "avg_session_uptime": 6389, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:39.755 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:39.756 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:39.757 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:39.758 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:39.759 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 301691, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 1723, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:39.760 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 173, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 17739176, 2024-04-25T19:06:39.761 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 124990, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "objecter": { 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "op_active": 0, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "op_laggy": 0, 2024-04-25T19:06:39.762 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send": 5088, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "op_send_bytes": 21, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "op_resend": 3, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "op_reply": 5088, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "op_latency": { 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5088, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 5.484839183, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.001077995 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.763 INFO:teuthology.orchestra.run.smithi070.stdout: "op_inflight": 0, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "oplen_avg": { 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 5088, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 5089 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "op": 5088, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "op_r": 5088, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "op_w": 0, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "op_rmw": 0, 2024-04-25T19:06:39.764 INFO:teuthology.orchestra.run.smithi070.stdout: "op_pg": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_stat": 2939, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_create": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_read": 2147, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_write": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writefull": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_writesame": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_append": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_zero": 0, 2024-04-25T19:06:39.765 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_truncate": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_delete": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_mapext": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_getxattr": 1, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_setxattr": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:39.766 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_call": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_watch": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_notify": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "osdop_other": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_active": 0, 2024-04-25T19:06:39.767 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_send": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_resend": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "linger_ping": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_active": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_send": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolop_resend": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_active": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_send": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "poolstat_resend": 0, 2024-04-25T19:06:39.768 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_active": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_send": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "statfs_resend": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "command_active": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "command_send": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "command_resend": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "map_epoch": 127, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "map_full": 0, 2024-04-25T19:06:39.769 INFO:teuthology.orchestra.run.smithi070.stdout: "map_inc": 35, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_sessions": 11, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_open": 12, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_session_close": 1, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_laggy": 0, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_wr": 0, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_rd": 2, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_del": 0 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.770 INFO:teuthology.orchestra.run.smithi070.stdout: "oft": { 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_objs": 0, 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_updates": 0, 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "omap_total_removes": 0 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "osd-slow-ops": { 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "slow_ops_count": 0 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.771 INFO:teuthology.orchestra.run.smithi070.stdout: "purge_queue": { 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executing_high_water": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed_ops": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_executed": 0, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.772 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 5576, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 391646733, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 5576, 2024-04-25T19:06:39.773 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5576, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 391646733, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.774 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:39.775 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5088, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 388911017, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5088, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 388911017, 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:39.776 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 1024, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:39.777 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 5088, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 5088, 2024-04-25T19:06:39.778 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 5088, 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 5088, 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:39.779 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:39.780 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:39.781 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-write_buf_throttle-0x564ac399e6e0": { 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 3758096384, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:39.782 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 0, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 0, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:39.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.786 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:39.787 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-25T19:06:40.489 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T19:06:40.489 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T19:06:40.490 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-04-25T19:06:40.490 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-04-25T19:06:40.490 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-25T19:06:40.490 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-25T19:06:40.490 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T19:06:40.490 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T19:06:40.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T19:06:40.727 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T19:06:40.849 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:40 smithi070 ceph-mon[32820]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 3.9 MiB/s wr, 148 op/s 2024-04-25T19:06:40.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:40 smithi063 ceph-mon[28006]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 3.9 MiB/s wr, 148 op/s 2024-04-25T19:06:41.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:40 smithi114 ceph-mon[32988]: pgmap v636: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 3.9 MiB/s wr, 148 op/s 2024-04-25T19:06:42.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:42 smithi070 ceph-mon[32820]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:42.869 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:42 smithi114 ceph-mon[32988]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:42.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:42 smithi063 ceph-mon[28006]: pgmap v637: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:42.991 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-25T19:06:42.991 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 0, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 2, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 0, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 0, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.001067278, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T19:06:42.992 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.000195420, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:42.993 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 2, 2024-04-25T19:06:42.994 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 306, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 320, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 209936, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 5, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.017316056, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.012875351, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.001159457, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:42.995 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 303, 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.045250442, 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000149341 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:42.996 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 209936 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_messages": 416, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages": 403, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_bytes": 1647248, 2024-04-25T19:06:42.997 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_bytes": 248544, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_total_time": 0.068629526, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_send_time": 0.023398429, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_recv_time": 0.042204487, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_running_fast_dispatch_time": 0.009256510, 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:42.998 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 403, 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.018300724, 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000045411 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:42.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_recv_encrypted_bytes": 1647248, 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "msgr_send_encrypted_bytes": 248544 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "cct": { 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "total_workers": 0, 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "unhealthy_workers": 0 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "mempool": { 2024-04-25T19:06:43.000 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:43.001 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:43.002 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:43.003 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_items": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:43.004 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_bytes": 148692, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_anon_items": 30, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_bytes": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_items": 0, 2024-04-25T19:06:43.005 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_items": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:43.006 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "pgmap_items": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_co_items": 2, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_1_items": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: "unittest_2_items": 0 2024-04-25T19:06:43.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "val": 0, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "max": 104857600, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "get_started": 0, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "get": 414, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "get_sum": 1586049, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:43.008 INFO:teuthology.orchestra.run.smithi070.stdout: "get_or_fail_success": 414, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "take": 0, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "take_sum": 0, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "put": 414, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "put_sum": 1586049, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "wait": { 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "avgcount": 0, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "sum": 0.000000000, 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: "avgtime": 0.000000000 2024-04-25T19:06:43.009 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:43.010 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-25T19:06:43.010 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-25T19:06:43.765 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-25T19:06:43.765 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-25T19:06:43.765 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-25T19:06:43.765 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-25T19:06:43.765 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-25T19:06:43.766 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-25T19:06:43.766 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-25T19:06:43.766 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T19:06:43.766 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T19:06:44.015 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T19:06:44.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:44 smithi114 ceph-mon[32988]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:44.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:44 smithi063 ceph-mon[28006]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:44.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:44 smithi070 ceph-mon[32820]: pgmap v638: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-04-25T19:06:45.808 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-25T19:06:45.808 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:45.808 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 286525, 2024-04-25T19:06:45.808 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 263194, 2024-04-25T19:06:45.808 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 82563457, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 203592203, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 11, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 14.638084582, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 6.353855126, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 9.982009333, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.479953913, 2024-04-25T19:06:45.809 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 263192, 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 7.583146292, 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000028812 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:45.810 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 516279, 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.078490503, 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000152 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:45.811 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 24697, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 24654, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 7965846, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 227966209, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 11, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:45.812 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 2.701282658, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 1.254115809, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.471732776, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.855272145, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 24653, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1.099597107, 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000044602 2024-04-25T19:06:45.813 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 21, 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000002745, 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000130 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1861664, 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 261712 2024-04-25T19:06:45.814 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 233401, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 337614, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 78277040, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 325979337, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 18, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 13.955499126, 2024-04-25T19:06:45.815 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 6.118653743, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 9.278853220, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.609208438, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 337611, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 11.445412709, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000033901 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.816 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 438982, 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.086679995, 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000197 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 7344, 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 774112 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.817 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 53957, 2024-04-25T19:06:45.818 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 77.998234909, 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001445562 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 35655, 2024-04-25T19:06:45.819 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 5.893241604, 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000165285 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 155066, 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 131878, 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-04-25T19:06:45.820 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 131878, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1098.042371353, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.008326198 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 23186, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 6, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 152, 2024-04-25T19:06:45.821 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 16, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 13, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 3837, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 1216, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 2207, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 414, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 2100, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 1134, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 203, 2024-04-25T19:06:45.822 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 203, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 12, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 265220, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 135217, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 23179, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 237, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 1, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:45.823 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 3712, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 7472, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 11, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 808, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 18, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 1309, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:45.824 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 754, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 212203978, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 8, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:45.825 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 36, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 15326, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 83886, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:45.826 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 48693, 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 18306, 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 7559, 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 101116 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 349, 2024-04-25T19:06:45.827 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 903, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 520, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 891, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 511, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 20190, 2024-04-25T19:06:45.828 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 20190, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-04-25T19:06:45.829 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_path": 13, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_inode": 1525, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 14, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 29, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:45.830 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 111307, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 24376, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 12291, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 99016, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-04-25T19:06:45.831 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 12085, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 158, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 30, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 16, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 142, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 9, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 14, 2024-04-25T19:06:45.832 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 21709817, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 431473945, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 0, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 29214, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 158.898539852, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005439123 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 0 2024-04-25T19:06:45.833 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1731, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 22744, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 21013, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 267, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 779, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 512, 2024-04-25T19:06:45.834 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 3837, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 38670, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 34833, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 203, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 23060, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 22857, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 87120, 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 223608 2024-04-25T19:06:45.835 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 227306, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 431, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 155066, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 3387, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 15028, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 20146, 2024-04-25T19:06:45.836 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 400.045684061, 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.019857325 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 3318, 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 32.885383162, 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.009911206 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:45.837 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.838 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 41240, 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 231.888952779, 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005622913 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.839 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-04-25T19:06:45.840 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.841 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.842 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:45.842 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.842 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.842 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.843 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 7, 2024-04-25T19:06:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.008291388, 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001184484 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 40875, 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 182.417884878, 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.004462822 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 5040, 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 43.477179579, 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.008626424 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.848 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.849 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 82, 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.833653620, 2024-04-25T19:06:45.850 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.010166507 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 980, 2024-04-25T19:06:45.851 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1.583552265, 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001615869 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.852 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.853 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 20190, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 204.901789621, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.010148677 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:45.854 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 1, 2024-04-25T19:06:45.855 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 4, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 3, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 1, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 27414, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 27414, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 1209, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.856 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:45.857 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:45.858 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:45.859 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:45.860 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 383029, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 1150, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 88, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 1, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-25T19:06:45.861 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 173, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:45.862 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 7611222, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 70620, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-25T19:06:45.863 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 52479, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 433063181, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 28, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 52479, 2024-04-25T19:06:45.864 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 52479, 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 295.872737200, 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005637926 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 52479, 2024-04-25T19:06:45.865 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 53771 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 52479, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 774, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 51705, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 0, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 21, 2024-04-25T19:06:45.866 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 768, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 29982, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 458, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 16, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 20377, 2024-04-25T19:06:45.867 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 2, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 42, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-04-25T19:06:45.868 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-04-25T19:06:45.869 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-04-25T19:06:45.870 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 127, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 35, 2024-04-25T19:06:45.871 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 12, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 12, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 1597, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 12, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 496 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.872 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 1, 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 1743 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-04-25T19:06:45.873 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 10, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 20190, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 20190, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.874 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 544619, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 119760557, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 544619, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:45.875 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 544619, 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 119760557, 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:45.876 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 52479, 2024-04-25T19:06:45.877 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 435105955, 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 52479, 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 435105955, 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.878 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:45.879 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 52479, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 52479, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 52479, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 52479, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.880 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 20190, 2024-04-25T19:06:45.881 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 2042870, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 20190, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 768, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 2042870, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.882 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x55fd32ae6500": { 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:45.883 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 111307, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 427279641, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 111307, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 29214, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 427279641, 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:45.884 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:45.885 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:45.885 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:45.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.885 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:45.885 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-25T19:06:46.538 INFO:tasks.check_counter:Daemon mds.c mds.exported=11 2024-04-25T19:06:46.538 INFO:tasks.check_counter:Daemon mds.c mds.imported=18 2024-04-25T19:06:46.538 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=349 2024-04-25T19:06:46.538 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=903 2024-04-25T19:06:46.538 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=8 2024-04-25T19:06:46.539 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-25T19:06:46.539 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T19:06:46.539 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T19:06:46.539 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T19:06:46.785 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T19:06:46.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:46 smithi114 ceph-mon[32988]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-25T19:06:46.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:46 smithi063 ceph-mon[28006]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-25T19:06:46.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:46 smithi070 ceph-mon[32820]: pgmap v639: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-04-25T19:06:48.561 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-25T19:06:48.561 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:48.561 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 346, 2024-04-25T19:06:48.561 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 337, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 17074146, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 120868, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 6, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 2, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.047406062, 2024-04-25T19:06:48.562 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.010211062, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.025787172, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.010204418, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 336, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.016420551, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000048870 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.563 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 208544, 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 34368 2024-04-25T19:06:48.564 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 211, 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 251, 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 35596205, 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 222693, 2024-04-25T19:06:48.565 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 6, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.037396367, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.006725179, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.023444224, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.005256064, 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:48.566 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 249, 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.012755110, 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000051225 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.567 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 155856 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 295, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 295, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 50410168, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 93002, 2024-04-25T19:06:48.568 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 6, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.046596382, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.006987189, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.030804483, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.007039456, 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:48.569 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 294, 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.015966023, 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000054306 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.570 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-25T19:06:48.571 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1255, 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.071110334, 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000056661 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.572 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:48.573 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:48.573 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:48.573 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.573 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.573 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 0, 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 0, 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.574 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 0, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 0, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:48.575 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 0, 2024-04-25T19:06:48.577 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 0, 2024-04-25T19:06:48.578 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 0, 2024-04-25T19:06:48.578 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 6440, 2024-04-25T19:06:48.578 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 3768, 2024-04-25T19:06:48.578 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 2672, 2024-04-25T19:06:48.578 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 1057, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 12, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 0, 2024-04-25T19:06:48.579 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 0, 2024-04-25T19:06:48.580 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 666, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 215009274, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 7, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:48.581 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 0, 2024-04-25T19:06:48.582 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 0, 2024-04-25T19:06:48.583 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 0 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-04-25T19:06:48.584 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:48.585 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_path": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_inode": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-04-25T19:06:48.586 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 0, 2024-04-25T19:06:48.587 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 24404, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 2, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 10, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-04-25T19:06:48.588 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 0, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 463597563, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 468647096, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 468637039, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.589 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 24404 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 977, 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 6382, 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 5405, 2024-04-25T19:06:48.590 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 109, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 176, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 67, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 6440, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 6773, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 333, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 0, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 0, 2024-04-25T19:06:48.591 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 0, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 60912, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 215392 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 0, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 0, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 0, 2024-04-25T19:06:48.592 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 0, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 0, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.593 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.594 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.595 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-04-25T19:06:48.596 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.597 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.598 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.599 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.600 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-04-25T19:06:48.600 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.600 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.602 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.603 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.604 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.605 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:48.606 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.607 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.608 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.609 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.610 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:48.611 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 2, 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 2, 2024-04-25T19:06:48.612 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 0, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 0, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 0, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 0, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 2939, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.613 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:48.614 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:48.615 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:48.616 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:48.617 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 269432, 2024-04-25T19:06:48.618 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 1331, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:48.619 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 173, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:48.620 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 7239271, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 53359, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.621 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 773, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 26, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 3, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 773, 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-04-25T19:06:48.622 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 773, 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1.070977432, 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001385481 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 773, 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 774 2024-04-25T19:06:48.623 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 773, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 773, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 0, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 434, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 0, 2024-04-25T19:06:48.624 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 336, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 0, 2024-04-25T19:06:48.625 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 2, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-04-25T19:06:48.626 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-04-25T19:06:48.627 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-04-25T19:06:48.628 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 127, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-04-25T19:06:48.629 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 4, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 9, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 9, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 0, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 2, 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 0 2024-04-25T19:06:48.630 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 0, 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-04-25T19:06:48.631 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 0 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 0, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 0, 2024-04-25T19:06:48.632 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 0, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 848, 2024-04-25T19:06:48.633 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 102997464, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 848, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 848, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 102997464, 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:48.634 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:48.635 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 773, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 102650692, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 773, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 102650692, 2024-04-25T19:06:48.636 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-04-25T19:06:48.637 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 773, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 773, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 773, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 773, 2024-04-25T19:06:48.638 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:48.639 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-04-25T19:06:48.640 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x55d9308bc6e0": { 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:48.641 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:48.642 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 0, 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 0, 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:48.643 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-25T19:06:48.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:48 smithi114 ceph-mon[32988]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-04-25T19:06:48.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:48 smithi063 ceph-mon[28006]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-04-25T19:06:48.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:48 smithi070 ceph-mon[32820]: pgmap v640: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 327 KiB/s wr, 32 op/s 2024-04-25T19:06:49.474 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T19:06:49.474 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T19:06:49.475 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-04-25T19:06:49.475 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-04-25T19:06:49.475 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=7 2024-04-25T19:06:49.475 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-25T19:06:49.475 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T19:06:49.475 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T19:06:49.476 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T19:06:49.716 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T19:06:50.805 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:50 smithi114 ceph-mon[32988]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 340 KiB/s wr, 35 op/s 2024-04-25T19:06:50.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:50 smithi063 ceph-mon[28006]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 340 KiB/s wr, 35 op/s 2024-04-25T19:06:50.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:50 smithi070 ceph-mon[32820]: pgmap v641: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 340 KiB/s wr, 35 op/s 2024-04-25T19:06:51.352 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-25T19:06:51.352 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:51.352 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 411859, 2024-04-25T19:06:51.352 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 476021, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 129017684, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 439625157, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 14, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 5, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 21.298139714, 2024-04-25T19:06:51.353 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 9.124522852, 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 15.285863940, 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.575235956, 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 476017, 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 15.202649188, 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000031937 2024-04-25T19:06:51.354 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 756212, 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.134785535, 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000178 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:51.355 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 98320, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 99015, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 25669745, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 119533661, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 18, 2024-04-25T19:06:51.356 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 4, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 6.804851327, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 3.288059843, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 3.565052687, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.523971839, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 99014, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 3.470577687, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000035051 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.357 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 164197, 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.028553004, 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000173 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 5168, 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 248112 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:51.358 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 16738, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 16938, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 7785450, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 167084060, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 7, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 6, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 1.733062011, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.683700371, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.330277429, 2024-04-25T19:06:51.359 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.630918021, 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 16937, 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.792969111, 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000046818 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:51.360 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1863408, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 784064 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 1, 2024-04-25T19:06:51.361 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-MDSRank": { 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 44787, 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 72.557234479, 2024-04-25T19:06:51.362 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.001620051 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "finisher-PurgeQueue": { 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "queue_len": 0, 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "complete_latency": { 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 32026, 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 5.115656261, 2024-04-25T19:06:51.363 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000159734 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "mds": { 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "request": 152086, 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "reply": 128732, 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "reply_latency": { 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 128732, 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1155.143972403, 2024-04-25T19:06:51.364 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.008973246 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_reply": 0, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "forward": 23353, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_complete": 12, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_fetch_keys": 0, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_commit": 159, 2024-04-25T19:06:51.365 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_split": 17, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_merge": 12, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes": 2960, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_top": 978, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_bottom": 1908, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pin_tail": 74, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_pinned": 1562, 2024-04-25T19:06:51.366 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_expired": 1890, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "inodes_with_caps": 143, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "caps": 143, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "subtrees": 12, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse": 239894, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_hit": 118859, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_forward": 23317, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_discover": 219, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_dir_fetch": 0, 2024-04-25T19:06:51.367 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_remote_ino": 0, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "traverse_lock": 2009, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "load_cent": 11617, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "q": 1, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "exported": 40, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "exported_inodes": 1388, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "imported": 16, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "imported_inodes": 1476, 2024-04-25T19:06:51.368 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_dir_fetch": 0, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_backtrace_fetch": 0, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "openino_peer_discover": 0, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rfiles": 754, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rbytes": 212203978, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "root_rsnaps": 8, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T19:06:51.369 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_set_tag": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_inotable_repaired": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_inodes": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "scrub_file_inodes": 0, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_inode_file_caps": 22, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_revoke": 17426, 2024-04-25T19:06:51.370 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_grant": 80316, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps": 48745, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_caps_dirty": 15584, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_cap_release": 7810, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "process_request_cap_release": 95765 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_cache": { 2024-04-25T19:06:51.371 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update": 313, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_update_receipt": 1052, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_try_discover": 595, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_send_discover": 1028, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "dir_handle_discover": 415, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays": 0, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_delayed": 0, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "num_strays_enqueuing": 0, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_created": 18214, 2024-04-25T19:06:51.372 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_enqueued": 18214, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_reintegrated": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "strays_migrated": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_processing": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_enqueued": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "num_recovering_prioritized": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_started": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "recovery_completed": 0, 2024-04-25T19:06:51.373 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_path": 13, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_quiesce_inode": 1550, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_exportdir": 50, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_flush": 0, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragmentdir": 29, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_fragstats": 0, 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: "ireq_inodestats": 0 2024-04-25T19:06:51.374 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_log": { 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evlrg": 0, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evadd": 103640, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evex": 14339, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evtrm": 12291, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "ev": 91349, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evexg": 0, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "evexd": 2048, 2024-04-25T19:06:51.375 INFO:teuthology.orchestra.run.smithi114.stdout: "segadd": 146, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "segex": 18, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "segtrm": 16, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "seg": 130, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "segmjr": 8, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "segexg": 0, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "segexd": 2, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "expos": 21029039, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "wrpos": 393180776, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "rdpos": 0, 2024-04-25T19:06:51.376 INFO:teuthology.orchestra.run.smithi114.stdout: "jlat": { 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 24562, 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 140.100979611, 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005703972 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "replayed": 0 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_mem": { 2024-04-25T19:06:51.377 INFO:teuthology.orchestra.run.smithi114.stdout: "ino": 1491, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "ino+": 21112, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "ino-": 19621, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dir": 250, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dir+": 833, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dir-": 583, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dn": 2960, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dn+": 40208, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "dn-": 37248, 2024-04-25T19:06:51.378 INFO:teuthology.orchestra.run.smithi114.stdout: "cap": 143, 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "cap+": 21730, 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "cap-": 21587, 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "rss": 88476, 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "heap": 223608 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.379 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_server": { 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_client_request": 212005, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "dispatch_server_request": 244, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_request": 152086, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_client_session": 2333, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "handle_peer_request": 9186, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "req_create_latency": { 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 18241, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 423.637029133, 2024-04-25T19:06:51.380 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.023224441 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getattr_latency": { 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 2857, 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 26.342185491, 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.009220225 2024-04-25T19:06:51.381 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getfilelock_latency": { 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "req_link_latency": { 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.382 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookup_latency": { 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 39113, 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 241.904896128, 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.006184769 2024-04-25T19:06:51.383 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookuphash_latency": { 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupino_latency": { 2024-04-25T19:06:51.384 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupname_latency": { 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.385 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupparent_latency": { 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lookupsnap_latency": { 2024-04-25T19:06:51.386 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.387 INFO:teuthology.orchestra.run.smithi114.stdout: "req_lssnap_latency": { 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mkdir_latency": { 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.388 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mknod_latency": { 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "req_mksnap_latency": { 2024-04-25T19:06:51.389 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "req_open_latency": { 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 1, 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.002401165, 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.002401165 2024-04-25T19:06:51.390 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "req_readdir_latency": { 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 46304, 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 184.499657659, 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.003984529 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rename_latency": { 2024-04-25T19:06:51.391 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 3458, 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 23.271276935, 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.006729692 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "req_renamesnap_latency": { 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.392 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "req_snapdiff_latency": { 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmdir_latency": { 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.393 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmsnap_latency": { 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.394 INFO:teuthology.orchestra.run.smithi114.stdout: "req_rmxattr_latency": { 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setattr_latency": { 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 76, 2024-04-25T19:06:51.395 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.715949195, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.009420384 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setdirlayout_latency": { 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setfilelock_latency": { 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 468, 2024-04-25T19:06:51.396 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 1.197288566, 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.002558308 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setlayout_latency": { 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.397 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "req_setxattr_latency": { 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "req_symlink_latency": { 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.398 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "req_unlink_latency": { 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 18214, 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 253.573288131, 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.013921889 2024-04-25T19:06:51.399 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_revoke_eviction": 0, 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "cap_acquisition_throttle": 0, 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "req_getvxattr_latency": { 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_sessions": { 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "session_count": 1, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "session_add": 3, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "session_remove": 2, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_open": 1, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "sessions_stale": 0, 2024-04-25T19:06:51.401 INFO:teuthology.orchestra.run.smithi114.stdout: "total_load": 26043, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "average_load": 26043, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "avg_session_uptime": 1129, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "mdthresh_evicted": 0 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:51.402 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:51.403 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:51.404 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:51.405 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:51.406 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 342877, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 810, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 176, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 2, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:51.407 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 8240, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 173, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-25T19:06:51.408 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 6271942, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 55312, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "objecter": { 2024-04-25T19:06:51.409 INFO:teuthology.orchestra.run.smithi114.stdout: "op_active": 0, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_laggy": 0, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send": 45560, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_send_bytes": 394292599, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_resend": 28, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_reply": 45560, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "op_latency": { 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 45560, 2024-04-25T19:06:51.410 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 267.693338164, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.005875621 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "op_inflight": 0, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "oplen_avg": { 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 45560, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 46786 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "op": 45560, 2024-04-25T19:06:51.411 INFO:teuthology.orchestra.run.smithi114.stdout: "op_r": 670, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "op_w": 44890, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "op_rmw": 0, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "op_pg": 0, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_stat": 0, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_create": 21, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_read": 658, 2024-04-25T19:06:51.412 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_write": 25220, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writefull": 413, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_writesame": 0, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_append": 0, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_zero": 0, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_truncate": 17, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_delete": 18396, 2024-04-25T19:06:51.413 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_mapext": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_sparse_read": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_clonerange": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_getxattr": 2, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_setxattr": 42, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_cmpxattr": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_rmxattr": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_resetxattrs": 0, 2024-04-25T19:06:51.414 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_call": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_watch": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_notify": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_pgls_filter": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "osdop_other": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_active": 0, 2024-04-25T19:06:51.415 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_send": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_resend": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "linger_ping": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_active": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_send": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "poolop_resend": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_active": 0, 2024-04-25T19:06:51.416 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_send": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "poolstat_resend": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_active": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_send": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "statfs_resend": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "command_active": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "command_send": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "command_resend": 0, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "map_epoch": 127, 2024-04-25T19:06:51.417 INFO:teuthology.orchestra.run.smithi114.stdout: "map_full": 0, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "map_inc": 35, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_sessions": 12, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_open": 12, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_session_close": 0, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_laggy": 0, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_wr": 1537, 2024-04-25T19:06:51.418 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_rd": 24, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_del": 456 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "oft": { 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_objs": 1, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_kv_pairs": 0, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_updates": 0, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "omap_total_removes": 0 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.419 INFO:teuthology.orchestra.run.smithi114.stdout: "osd-slow-ops": { 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "slow_ops_count": 1769 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "purge_queue": { 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops": 0, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing": 0, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executing_high_water": 10, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed_ops": 18214, 2024-04-25T19:06:51.420 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_executed": 18214, 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "pq_item_in_journal": 0 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:51.421 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 526913, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 115021166, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 526913, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 526913, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 115021166, 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:51.422 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_bytes": { 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:51.423 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 45560, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 396135517, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 45560, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 396135517, 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:51.424 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-objecter_ops": { 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 1024, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:51.425 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 0, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 0, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 0, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 45560, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 45560, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 45560, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 45560, 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:51.426 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle": { 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:51.427 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 18214, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1843110, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 18214, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:51.428 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 658, 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1843110, 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.429 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-write_buf_throttle-0x56451951e500": { 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 3758096384, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 103640, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 388986472, 2024-04-25T19:06:51.430 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 103640, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 24562, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 388986472, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:51.431 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.432 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:51.432 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-25T19:06:52.100 INFO:tasks.check_counter:Daemon mds.i mds.exported=40 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds.imported=16 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=313 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1052 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=8 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-25T19:06:52.101 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-25T19:06:52.101 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T19:06:52.102 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T19:06:52.340 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T19:06:52.811 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:52 smithi114 ceph-mon[32988]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:52.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:52 smithi063 ceph-mon[28006]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:52.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:52 smithi070 ceph-mon[32820]: pgmap v642: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:53.951 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2024-04-25T19:06:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T19:06:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 0, 2024-04-25T19:06:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 2, 2024-04-25T19:06:53.951 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 0, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 0, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 4, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 0, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.000923353, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T19:06:53.952 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.000135195, 2024-04-25T19:06:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:53.953 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T19:06:53.954 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 3, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 311, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 480, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 367424, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 7, 2024-04-25T19:06:53.955 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 3, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.017914832, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.012702227, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.001837923, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 308, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.045716538, 2024-04-25T19:06:53.956 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000148430 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 367424 2024-04-25T19:06:53.957 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_messages": 420, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages": 407, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_bytes": 1648336, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_bytes": 251104, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_created_connections": 2, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_active_connections": 1, 2024-04-25T19:06:53.958 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_total_time": 0.069342102, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_send_time": 0.023642190, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_recv_time": 0.042661773, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_running_fast_dispatch_time": 0.009432200, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 407, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.017686489, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000043455 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_handle_ack_lat": { 2024-04-25T19:06:53.959 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_recv_encrypted_bytes": 1648336, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "msgr_send_encrypted_bytes": 251104 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "cct": { 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "total_workers": 0, 2024-04-25T19:06:53.960 INFO:teuthology.orchestra.run.smithi114.stdout: "unhealthy_workers": 0 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "mempool": { 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_bytes": 0, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bloom_filter_items": 0, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_alloc_items": 0, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T19:06:53.961 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_data_items": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_other_items": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T19:06:53.962 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_bytes": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_extent_items": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_bytes": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_blob_items": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T19:06:53.963 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_fsck_items": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_bytes": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_txc_items": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_bytes": 0, 2024-04-25T19:06:53.964 INFO:teuthology.orchestra.run.smithi114.stdout: "bluestore_writing_items": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_bytes": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_items": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_reader_items": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "bluefs_file_writer_items": 0, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_bytes": 148692, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_anon_items": 30, 2024-04-25T19:06:53.965 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_bytes": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "buffer_meta_items": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_bytes": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_items": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_bytes": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_mapbl_items": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_bytes": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osd_pglog_items": 0, 2024-04-25T19:06:53.966 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_bytes": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_items": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "osdmap_mapping_items": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_bytes": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "pgmap_items": 0, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_bytes": 928, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "mds_co_items": 2, 2024-04-25T19:06:53.967 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_bytes": 0, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_1_items": 0, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_bytes": 0, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "unittest_2_items": 0 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: }, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "val": 0, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "max": 104857600, 2024-04-25T19:06:53.968 INFO:teuthology.orchestra.run.smithi114.stdout: "get_started": 0, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get": 419, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get_sum": 1586596, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_fail": 0, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "get_or_fail_success": 419, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "take": 0, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "take_sum": 0, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "put": 419, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "put_sum": 1586596, 2024-04-25T19:06:53.969 INFO:teuthology.orchestra.run.smithi114.stdout: "wait": { 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "avgcount": 0, 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "sum": 0.000000000, 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: "avgtime": 0.000000000 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout: } 2024-04-25T19:06:53.970 INFO:teuthology.orchestra.run.smithi114.stdout:} 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-25T19:06:54.609 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-25T19:06:54.610 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-25T19:06:54.610 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T19:06:54.622 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T19:06:54.622 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T19:06:54.623 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:06:54.685 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T19:06:54.685 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:06:54.740 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd blocklist ls 2024-04-25T19:06:54.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:54 smithi114 ceph-mon[32988]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:54.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:54 smithi063 ceph-mon[28006]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:54.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:54 smithi070 ceph-mon[32820]: pgmap v643: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2024-04-25T19:06:55.211 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:06:56.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:56 smithi063 ceph-mon[28006]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-04-25T19:06:56.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:56 smithi070 ceph-mon[32820]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-04-25T19:06:57.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:56 smithi114 ceph-mon[32988]: pgmap v644: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 83 KiB/s wr, 13 op/s 2024-04-25T19:06:57.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 -- 172.21.15.63:0/2499808332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 msgr2=0x7f2374108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 --2- 172.21.15.63:0/2499808332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f2374108e10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2364009990 tx=0x7f2364009ca0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 -- 172.21.15.63:0/2499808332 shutdown_connections 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 --2- 172.21.15.63:0/2499808332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406b0f0 0x7f237410f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 --2- 172.21.15.63:0/2499808332 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f2374108e10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 --2- 172.21.15.63:0/2499808332 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406a750 0x7f237406abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 -- 172.21.15.63:0/2499808332 >> 172.21.15.63:0/2499808332 conn(0x7f23740788f0 msgr2=0x7f2374078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:57.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 -- 172.21.15.63:0/2499808332 shutdown_connections 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.384+0000 7f2378e00700 1 -- 172.21.15.63:0/2499808332 wait complete. 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.385+0000 7f2378e00700 1 Processor -- start 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.385+0000 7f2378e00700 1 -- start start 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 0x7f2374109910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 0x7f2374109e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f237410a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2374112c20 con 0x7f237406a750 2024-04-25T19:06:57.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2374112aa0 con 0x7f2374108a30 2024-04-25T19:06:57.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2378e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f2374112da0 con 0x7f237406b0f0 2024-04-25T19:06:57.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2371e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 0x7f2374109910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:57.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2371e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 0x7f2374109910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38724/0 (socket says 172.21.15.63:38724) 2024-04-25T19:06:57.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2371400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 0x7f2374109e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:57.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2371e00700 1 -- 172.21.15.63:0/1177280772 learned_addr learned my addr 172.21.15.63:0/1177280772 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:06:57.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.386+0000 7f2372800700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f237410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:57.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 -- 172.21.15.63:0/1177280772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 msgr2=0x7f237410a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:57.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f237410a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 -- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 msgr2=0x7f2374109910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 0x7f2374109910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 -- 172.21.15.63:0/1177280772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23741abaa0 con 0x7f237406b0f0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2371400700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 0x7f2374109e50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f236000ea20 tx=0x7f236000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236000cbd0 con 0x7f237406b0f0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23600044d0 con 0x7f237406b0f0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360005020 con 0x7f237406b0f0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.387+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23741abd90 con 0x7f237406b0f0 2024-04-25T19:06:57.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.388+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23741ac270 con 0x7f237406b0f0 2024-04-25T19:06:57.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.389+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2374104170 con 0x7f237406b0f0 2024-04-25T19:06:57.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.391+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2360004750 con 0x7f237406b0f0 2024-04-25T19:06:57.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.392+0000 7f236aa00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 0x7f23580a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:06:57.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.393+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8587+0+0 (secure 0 0 0) 0x7f23600d07a0 con 0x7f237406b0f0 2024-04-25T19:06:57.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.393+0000 7f2371e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 0x7f23580a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:06:57.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.394+0000 7f2371e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 0x7f23580a62f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23640098e0 tx=0x7f236400b2a0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:06:57.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.397+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f236009cda0 con 0x7f237406b0f0 2024-04-25T19:06:57.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.648+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2374069730 con 0x7f237406b0f0 2024-04-25T19:06:57.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.649+0000 7f236aa00700 1 -- 172.21.15.63:0/1177280772 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v127) ==== 81+0+945 (secure 0 0 0) 0x7f2374069730 con 0x7f237406b0f0 2024-04-25T19:06:57.651 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3012688827 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3560456479 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/2327478809 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/3659595029 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/3403991527 2024-04-26T18:33:52.960272+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3648036576 2024-04-26T18:33:52.960272+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1446926443 2024-04-26T18:33:52.960272+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/3403991527 2024-04-26T18:33:52.960272+0000 2024-04-25T19:06:57.652 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/2210114679 2024-04-26T18:33:16.955897+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6801/3659595029 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:6800/2210114679 2024-04-26T18:33:16.955897+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/4120416918 2024-04-26T18:33:16.955897+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1747359887 2024-04-26T18:33:16.955897+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/3611634789 2024-04-26T18:46:32.881015+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/1119313138 2024-04-26T18:33:16.955897+0000 2024-04-25T19:06:57.653 INFO:teuthology.orchestra.run.smithi063.stdout:172.21.15.63:0/434371372 2024-04-26T18:33:52.960272+0000 2024-04-25T19:06:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.652+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 msgr2=0x7f23580a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.652+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 0x7f23580a62f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f23640098e0 tx=0x7f236400b2a0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 msgr2=0x7f2374109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:06:57.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 0x7f2374109e50 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f236000ea20 tx=0x7f236000ed30 comp rx=0 tx=0).stop 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 shutdown_connections 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f23580a3e30 0x7f23580a62f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2374108a30 0x7f237410a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f237406a750 0x7f2374109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 --2- 172.21.15.63:0/1177280772 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f237406b0f0 0x7f2374109e50 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 >> 172.21.15.63:0/1177280772 conn(0x7f23740788f0 msgr2=0x7f23740ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:06:57.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 shutdown_connections 2024-04-25T19:06:57.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:06:57.653+0000 7f2378e00700 1 -- 172.21.15.63:0/1177280772 wait complete. 2024-04-25T19:06:57.656 INFO:teuthology.orchestra.run.smithi063.stderr:listed 16 entries 2024-04-25T19:06:58.421 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T19:06:58.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:58.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T19:06:58.737 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T19:06:58.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:58.738 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T19:06:58.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:58.841 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-04-25T19:06:58.919 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:06:58.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:58.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:06:58.953 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:58 smithi070 ceph-mon[32820]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 7 op/s 2024-04-25T19:06:58.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:06:58 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/1177280772' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:06:58.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:58 smithi114 ceph-mon[32988]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 7 op/s 2024-04-25T19:06:58.964 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:06:58 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/1177280772' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:06:59.007 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:58 smithi063 ceph-mon[28006]: pgmap v645: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 KiB/s wr, 7 op/s 2024-04-25T19:06:59.008 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:06:58 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/1177280772' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T19:06:59.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:59.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link delete ceph-brx 2024-04-25T19:06:59.124 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T19:06:59.125 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T19:06:59.125 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:06:59.163 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052737,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/ca984ab377fe0066894453f802798e47359696f280cae8c63336971933833f61/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/ca984ab377fe0066894453f802798e47359696f280cae8c63336971933833f61/diff,workdir=/var/lib/containers/storage/overlay/ca984ab377fe0066894453f802798e47359696f280cae8c63336971933833f61/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1f5b88aace12cc3c5b1d6281698a92dfa2e7e4a88647c250f390ed35f13637ba/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/1f5b88aace12cc3c5b1d6281698a92dfa2e7e4a88647c250f390ed35f13637ba/diff,workdir=/var/lib/containers/storage/overlay/1f5b88aace12cc3c5b1d6281698a92dfa2e7e4a88647c250f390ed35f13637ba/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249004k,nr_inodes=812251,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/39d53902dec7152704a1bd26bd82fa4c78c698fabcaf9d8b07d8dc186618d73e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/39d53902dec7152704a1bd26bd82fa4c78c698fabcaf9d8b07d8dc186618d73e/diff,workdir=/var/lib/containers/storage/overlay/39d53902dec7152704a1bd26bd82fa4c78c698fabcaf9d8b07d8dc186618d73e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b57d96d7c79fe8c73386efd452ab579a492feefce62934452302033a910c9304/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/b57d96d7c79fe8c73386efd452ab579a492feefce62934452302033a910c9304/diff,workdir=/var/lib/containers/storage/overlay/b57d96d7c79fe8c73386efd452ab579a492feefce62934452302033a910c9304/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4a21963442cb6537f13e06fd3753acf1995de4c5fea082efc9470dd250e0bd21/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/4a21963442cb6537f13e06fd3753acf1995de4c5fea082efc9470dd250e0bd21/diff,workdir=/var/lib/containers/storage/overlay/4a21963442cb6537f13e06fd3753acf1995de4c5fea082efc9470dd250e0bd21/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ce07e4769a306c5f8e55947f95e0d309a2aa36006b3e3f4780e62df1c5f2a2ae/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/ce07e4769a306c5f8e55947f95e0d309a2aa36006b3e3f4780e62df1c5f2a2ae/diff,workdir=/var/lib/containers/storage/overlay/ce07e4769a306c5f8e55947f95e0d309a2aa36006b3e3f4780e62df1c5f2a2ae/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5b9c417b635906fa03949ccb1035e2a76a2e498f75bb0a1ff09520fb42c876ce/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/5b9c417b635906fa03949ccb1035e2a76a2e498f75bb0a1ff09520fb42c876ce/diff,workdir=/var/lib/containers/storage/overlay/5b9c417b635906fa03949ccb1035e2a76a2e498f75bb0a1ff09520fb42c876ce/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/646a3c37484796072de1cb533259ee546dd76f298f18c9b694dc28447db70282/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/646a3c37484796072de1cb533259ee546dd76f298f18c9b694dc28447db70282/diff,workdir=/var/lib/containers/storage/overlay/646a3c37484796072de1cb533259ee546dd76f298f18c9b694dc28447db70282/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f85297f449c765e1f34bc3b5ae877d39672efbd1aaeb10bf9986788c84ddb692/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/f85297f449c765e1f34bc3b5ae877d39672efbd1aaeb10bf9986788c84ddb692/diff,workdir=/var/lib/containers/storage/overlay/f85297f449c765e1f34bc3b5ae877d39672efbd1aaeb10bf9986788c84ddb692/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0fc3e20955e585d05454a84e66d537a9154a2b8bd2ca5250987ff91c2688a309/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WCPZSIQVTOIYRSS7HDVDXPH3LL:/var/lib/containers/storage/overlay/l/CB46OBVKBZD3S5MKEGY66TGQSL,upperdir=/var/lib/containers/storage/overlay/0fc3e20955e585d05454a84e66d537a9154a2b8bd2ca5250987ff91c2688a309/diff,workdir=/var/lib/containers/storage/overlay/0fc3e20955e585d05454a84e66d537a9154a2b8bd2ca5250987ff91c2688a309/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498008k,nr_inodes=819200,mode=755 0 0 2024-04-25T19:06:59.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:59.163 DEBUG:teuthology.orchestra.run.smithi063:> ip netns list 2024-04-25T19:06:59.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:06:59.251 DEBUG:teuthology.orchestra.run.smithi063:> sudo ip link delete ceph-brx 2024-04-25T19:06:59.389 INFO:teuthology.orchestra.run.smithi063.stderr:Cannot find device "ceph-brx" 2024-04-25T19:06:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:06:59.393 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T19:06:59.405 INFO:tasks.cephadm:Teardown begin 2024-04-25T19:06:59.405 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:06:59.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:06:59.842 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:00.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:00 smithi063 ceph-mon[28006]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 127 KiB/s wr, 12 op/s 2024-04-25T19:07:00.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:00 smithi070 ceph-mon[32820]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 127 KiB/s wr, 12 op/s 2024-04-25T19:07:01.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:00 smithi114 ceph-mon[32988]: pgmap v646: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 127 KiB/s wr, 12 op/s 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.074+0000 7f905be00700 1 -- 172.21.15.63:0/512414706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90541092d0 msgr2=0x7f90541096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.074+0000 7f905be00700 1 --2- 172.21.15.63:0/512414706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90541092d0 0x7f90541096b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f90480099c0 tx=0x7f9048009cd0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 -- 172.21.15.63:0/512414706 shutdown_connections 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 --2- 172.21.15.63:0/512414706 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f905410e3b0 0x7f9054115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 --2- 172.21.15.63:0/512414706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90541092d0 0x7f90541096b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 --2- 172.21.15.63:0/512414706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9054109c80 0x7f905410dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 -- 172.21.15.63:0/512414706 >> 172.21.15.63:0/512414706 conn(0x7f9054076210 msgr2=0x7f9054107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:02.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 -- 172.21.15.63:0/512414706 shutdown_connections 2024-04-25T19:07:02.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.075+0000 7f905be00700 1 -- 172.21.15.63:0/512414706 wait complete. 2024-04-25T19:07:02.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.076+0000 7f905be00700 1 Processor -- start 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.076+0000 7f905be00700 1 -- start start 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.076+0000 7f905be00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f905be00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9054109c80 0x7f90541a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f905be00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f905410e3b0 0x7f905419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f905be00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9054119290 con 0x7f905410e3b0 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f905be00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9054119110 con 0x7f90541092d0 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f905be00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9054119410 con 0x7f9054109c80 2024-04-25T19:07:02.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f905410e3b0 0x7f905419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:42552/0 (socket says 172.21.15.63:42552) 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 -- 172.21.15.63:0/2580704208 learned_addr learned my addr 172.21.15.63:0/2580704208 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9058a00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9054109c80 0x7f90541a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 -- 172.21.15.63:0/2580704208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9054109c80 msgr2=0x7f90541a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9054109c80 0x7f90541a5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 -- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f905410e3b0 msgr2=0x7f905419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:02.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f905410e3b0 0x7f905419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.077+0000 7f9059400700 1 -- 172.21.15.63:0/2580704208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905419fd10 con 0x7f90541092d0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f9059400700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9048006010 tx=0x7f90480055f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048026070 con 0x7f90541092d0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f904800f460 con 0x7f90541092d0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905419ffa0 con 0x7f90541092d0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9048009e60 con 0x7f90541092d0 2024-04-25T19:07:02.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.078+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90541ac2c0 con 0x7f90541092d0 2024-04-25T19:07:02.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.079+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9054069730 con 0x7f90541092d0 2024-04-25T19:07:02.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.081+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9048029070 con 0x7f90541092d0 2024-04-25T19:07:02.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.083+0000 7f9052000700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 0x7f903c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:02.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.083+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 8587+0+0 (secure 0 0 0) 0x7f90480d1420 con 0x7f90541092d0 2024-04-25T19:07:02.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.083+0000 7f9058a00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 0x7f903c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:02.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.084+0000 7f9058a00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 0x7f903c0a6270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9054108830 tx=0x7f9040009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:02.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.085+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9048030020 con 0x7f90541092d0 2024-04-25T19:07:02.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.364+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f905402df80 con 0x7f90541092d0 2024-04-25T19:07:02.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.365+0000 7f9052000700 1 -- 172.21.15.63:0/2580704208 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11874 (secure 0 0 0) 0x7f9048013070 con 0x7f90541092d0 2024-04-25T19:07:02.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:07:02.367 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:05:51.195626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T19:07:02.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 msgr2=0x7f903c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 0x7f903c0a6270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9054108830 tx=0x7f9040009260 comp rx=0 tx=0).stop 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 msgr2=0x7f90541a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9048006010 tx=0x7f90480055f0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 shutdown_connections 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f903c0a3db0 0x7f903c0a6270 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f905410e3b0 0x7f905419f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f90541092d0 0x7f90541a53f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 --2- 172.21.15.63:0/2580704208 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9054109c80 0x7f90541a5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:02.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.369+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 >> 172.21.15.63:0/2580704208 conn(0x7f9054076210 msgr2=0x7f9054077e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:02.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.370+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 shutdown_connections 2024-04-25T19:07:02.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:02.370+0000 7f905be00700 1 -- 172.21.15.63:0/2580704208 wait complete. 2024-04-25T19:07:02.372 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 62 2024-04-25T19:07:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:02 smithi070 ceph-mon[32820]: pgmap v647: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:02.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:02 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2580704208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:03.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:02 smithi063 ceph-mon[28006]: pgmap v647: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:03.020 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:02 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2580704208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:02 smithi114 ceph-mon[32988]: pgmap v647: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:03.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:02 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2580704208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:03.105 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs dump --format=json 2024-04-25T19:07:03.582 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:04.801 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:04 smithi063 ceph-mon[28006]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:04.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:04 smithi070 ceph-mon[32820]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:05.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:04 smithi114 ceph-mon[32988]: pgmap v648: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 113 KiB/s wr, 8 op/s 2024-04-25T19:07:05.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.747+0000 7f47c8800700 1 -- 172.21.15.63:0/247779345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c01034b0 msgr2=0x7f47c010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:05.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.747+0000 7f47c8800700 1 --2- 172.21.15.63:0/247779345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c01034b0 0x7f47c010f920 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009fc0 tx=0x7f47bc00b330 comp rx=0 tx=0).stop 2024-04-25T19:07:05.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.748+0000 7f47c8800700 1 -- 172.21.15.63:0/247779345 shutdown_connections 2024-04-25T19:07:05.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.748+0000 7f47c8800700 1 --2- 172.21.15.63:0/247779345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c01034b0 0x7f47c010f920 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:05.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.748+0000 7f47c8800700 1 --2- 172.21.15.63:0/247779345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c0107940 0x7f47c0107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:05.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.748+0000 7f47c8800700 1 --2- 172.21.15.63:0/247779345 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0102af0 0x7f47c0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:05.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.748+0000 7f47c8800700 1 -- 172.21.15.63:0/247779345 >> 172.21.15.63:0/247779345 conn(0x7f47c0079ba0 msgr2=0x7f47c0079fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:05.751 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:05 smithi063 ceph-mon[28006]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T19:07:05.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.750+0000 7f47c8800700 1 -- 172.21.15.63:0/247779345 shutdown_connections 2024-04-25T19:07:05.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.751+0000 7f47c8800700 1 -- 172.21.15.63:0/247779345 wait complete. 2024-04-25T19:07:05.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.752+0000 7f47c8800700 1 Processor -- start 2024-04-25T19:07:05.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 -- start start 2024-04-25T19:07:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 0x7f47c0199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c01034b0 0x7f47c0199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47c0112c90 con 0x7f47c0102af0 2024-04-25T19:07:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f47c0112b10 con 0x7f47c01034b0 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.753+0000 7f47c8800700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f47c0112e10 con 0x7f47c0107940 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 0x7f47c0199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c6800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c6800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:41506/0 (socket says 172.21.15.63:41506) 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c6800700 1 -- 172.21.15.63:0/597789372 learned_addr learned my addr 172.21.15.63:0/597789372 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c5400700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c01034b0 0x7f47c0199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.754+0000 7f47c5e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 0x7f47c0199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38768/0 (socket says 172.21.15.63:38768) 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 -- 172.21.15.63:0/597789372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c01034b0 msgr2=0x7f47c0199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c01034b0 0x7f47c0199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 -- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 msgr2=0x7f47c0199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 0x7f47c0199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 -- 172.21.15.63:0/597789372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c019de10 con 0x7f47c0107940 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.755+0000 7f47c6800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009790 tx=0x7f47bc00cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:05.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.756+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc021070 con 0x7f47c0107940 2024-04-25T19:07:05.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.756+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47bc007ac0 con 0x7f47c0107940 2024-04-25T19:07:05.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.756+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0052a0 con 0x7f47c0107940 2024-04-25T19:07:05.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.756+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c019e0a0 con 0x7f47c0107940 2024-04-25T19:07:05.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.756+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c019e590 con 0x7f47c0107940 2024-04-25T19:07:05.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.759+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c0069730 con 0x7f47c0107940 2024-04-25T19:07:05.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.762+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f47bc024070 con 0x7f47c0107940 2024-04-25T19:07:05.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.764+0000 7f47baa00700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 0x7f47a80a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:05.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.765+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8631+0+0 (secure 0 0 0) 0x7f47bc0d17d0 con 0x7f47c0107940 2024-04-25T19:07:05.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.765+0000 7f47c5e00700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 0x7f47a80a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:05.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.767+0000 7f47c5e00700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 0x7f47a80a62f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f47b400abd0 tx=0x7f47b400a380 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:05.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:05.769+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47bc09de90 con 0x7f47c0107940 2024-04-25T19:07:05.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:05 smithi070 ceph-mon[32820]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T19:07:06.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:05 smithi114 ceph-mon[32988]: osdmap e128: 12 total, 12 up, 12 in 2024-04-25T19:07:06.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.241+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47c01a3b70 con 0x7f47c0107940 2024-04-25T19:07:06.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.242+0000 7f47baa00700 1 -- 172.21.15.63:0/597789372 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+11874 (secure 0 0 0) 0x7f47bc09d790 con 0x7f47c0107940 2024-04-25T19:07:06.244 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:07:06.244 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14391,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/3937073235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":3937073235},{"type":"v1","addr":"172.21.15.63:6839","nonce":3937073235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6841/512888768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6840","nonce":512888768},{"type":"v1","addr":"172.21.15.63:6841","nonce":512888768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2179103180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2179103180},{"type":"v1","addr":"172.21.15.70:6833","nonce":2179103180}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.114:6833/1215662020","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6832","nonce":1215662020},{"type":"v1","addr":"172.21.15.114:6833","nonce":1215662020}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/3714473913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":3714473913},{"type":"v1","addr":"172.21.15.63:6835","nonce":3714473913}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/166123931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":166123931},{"type":"v1","addr":"172.21.15.63:6837","nonce":166123931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":62,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T18:42:31.925526+0000","modified":"2024-04-25T19:05:51.195626+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24533,"mds_1":24520,"mds_2":24508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24473":{"gid":24473,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/3182056159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":3182056159},{"type":"v1","addr":"172.21.15.70:6835","nonce":3182056159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6837/4043589496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":4043589496},{"type":"v1","addr":"172.21.15.70:6837","nonce":4043589496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.114:6837/1526734616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6836","nonce":1526734616},{"type":"v1","addr":"172.21.15.114:6837","nonce":1526734616}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24520":{"gid":24520,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":15,"addr":"172.21.15.114:6839/2417088150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6838","nonce":2417088150},{"type":"v1","addr":"172.21.15.114:6839","nonce":2417088150}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.70:6839/1944035005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":1944035005},{"type":"v1","addr":"172.21.15.70:6839","nonce":1944035005}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27178":{"gid":27178,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.114:6835/198947212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6834","nonce":198947212},{"type":"v1","addr":"172.21.15.114:6835","nonce":198947212}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24533,"qdb_cluster":[24533,24508,24520]},"id":1}]} 2024-04-25T19:07:06.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 msgr2=0x7f47a80a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:06.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 0x7f47a80a62f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f47b400abd0 tx=0x7f47b400a380 comp rx=0 tx=0).stop 2024-04-25T19:07:06.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 msgr2=0x7f47c019d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:06.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009790 tx=0x7f47bc00cc30 comp rx=0 tx=0).stop 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 shutdown_connections 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f47a80a3e30 0x7f47a80a62f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f47c0107940 0x7f47c019d8d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47c0102af0 0x7f47c0199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 --2- 172.21.15.63:0/597789372 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f47c01034b0 0x7f47c0199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 >> 172.21.15.63:0/597789372 conn(0x7f47c0079ba0 msgr2=0x7f47c0100a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 shutdown_connections 2024-04-25T19:07:06.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:06.244+0000 7f47c8800700 1 -- 172.21.15.63:0/597789372 wait complete. 2024-04-25T19:07:06.248 INFO:teuthology.orchestra.run.smithi063.stderr:dumped fsmap epoch 62 2024-04-25T19:07:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:06 smithi070 ceph-mon[32820]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:06.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:06 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/597789372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:06 smithi114 ceph-mon[32988]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:07.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:06 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/597789372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:07.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:06 smithi063 ceph-mon[28006]: pgmap v650: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:07.128 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:06 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/597789372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T19:07:07.253 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T19:07:07.710 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:08.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:08 smithi070 ceph-mon[32820]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:08.981 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:08 smithi063 ceph-mon[28006]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:09.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:08 smithi114 ceph-mon[32988]: pgmap v651: 97 pgs: 97 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 30 op/s 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 -- 172.21.15.63:0/2463859115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5920105480 msgr2=0x7f5920105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 --2- 172.21.15.63:0/2463859115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5920105480 0x7f5920105910 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5918009f90 tx=0x7f591800b330 comp rx=0 tx=0).stop 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 -- 172.21.15.63:0/2463859115 shutdown_connections 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 --2- 172.21.15.63:0/2463859115 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f592006a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 --2- 172.21.15.63:0/2463859115 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5920105480 0x7f5920105910 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 --2- 172.21.15.63:0/2463859115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f592006ac90 0x7f592006b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 -- 172.21.15.63:0/2463859115 >> 172.21.15.63:0/2463859115 conn(0x7f59200788f0 msgr2=0x7f5920078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:10.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 -- 172.21.15.63:0/2463859115 shutdown_connections 2024-04-25T19:07:10.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.163+0000 7f5925e00700 1 -- 172.21.15.63:0/2463859115 wait complete. 2024-04-25T19:07:10.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 Processor -- start 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 -- start start 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920105480 0x7f592019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f59201a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5920119230 con 0x7f5920105480 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59201190b0 con 0x7f592006ac90 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.165+0000 7f5925e00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f59201193b0 con 0x7f5920107970 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:10.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f5916000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920105480 0x7f592019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591f400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f59201a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33422/0 (socket says 172.21.15.63:33422) 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 -- 172.21.15.63:0/2407513602 learned_addr learned my addr 172.21.15.63:0/2407513602 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 -- 172.21.15.63:0/2407513602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 msgr2=0x7f59201a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f59201a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 -- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920105480 msgr2=0x7f592019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920105480 0x7f592019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591ea00700 1 -- 172.21.15.63:0/2407513602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59201a4730 con 0x7f592006ac90 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.166+0000 7f591f400700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f59201a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f591ea00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f591000e9f0 tx=0x7f591000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910004a30 con 0x7f592006ac90 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f591000cc70 con 0x7f592006ac90 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59201a4a20 con 0x7f592006ac90 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910005540 con 0x7f592006ac90 2024-04-25T19:07:10.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.167+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59201ac2a0 con 0x7f592006ac90 2024-04-25T19:07:10.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.168+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5920069730 con 0x7f592006ac90 2024-04-25T19:07:10.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.170+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5910004bd0 con 0x7f592006ac90 2024-04-25T19:07:10.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.172+0000 7f5924a00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 0x7f59040a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:10.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.172+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8631+0+0 (secure 0 0 0) 0x7f59100d0bc0 con 0x7f592006ac90 2024-04-25T19:07:10.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.172+0000 7f5916000700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 0x7f59040a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:10.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.173+0000 7f5916000700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 0x7f59040a6270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5908009680 tx=0x7f5908009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.174+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5910053ff0 con 0x7f592006ac90 2024-04-25T19:07:10.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.408+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59201ac470 con 0x7f592006ac90 2024-04-25T19:07:10.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.410+0000 7f5924a00700 1 -- 172.21.15.63:0/2407513602 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) ==== 74+0+23276 (secure 0 0 0) 0x7f591000ce20 con 0x7f592006ac90 2024-04-25T19:07:10.412 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T19:07:10.413 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":128,"fsid":"e90c993c-0331-11ef-bc93-c7b262605968","created":"2024-04-25T18:32:41.220967+0000","modified":"2024-04-25T19:07:04.641356+0000","last_up_change":"2024-04-25T18:41:09.924336+0000","last_in_change":"2024-04-25T18:40:49.037778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T18:36:47.142075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T18:42:23.312145+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T18:42:27.756785+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":36,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0615248c-d2e1-48aa-ab67-0733aaa339cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6803","nonce":237733214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6805","nonce":237733214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6809","nonce":237733214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":237733214},{"type":"v1","addr":"172.21.15.63:6807","nonce":237733214}]},"public_addr":"172.21.15.63:6803/237733214","cluster_addr":"172.21.15.63:6805/237733214","heartbeat_back_addr":"172.21.15.63:6809/237733214","heartbeat_front_addr":"172.21.15.63:6807/237733214","state":["exists","up"]},{"osd":1,"uuid":"2c5dd2fb-256f-461a-aa35-959c06ed8333","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6801","nonce":135943968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6803","nonce":135943968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6807","nonce":135943968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":135943968},{"type":"v1","addr":"172.21.15.70:6805","nonce":135943968}]},"public_addr":"172.21.15.70:6801/135943968","cluster_addr":"172.21.15.70:6803/135943968","heartbeat_back_addr":"172.21.15.70:6807/135943968","heartbeat_front_addr":"172.21.15.70:6805/135943968","state":["exists","up"]},{"osd":2,"uuid":"719d6679-f15c-4e02-b8c9-e19a26526679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6801","nonce":3090454830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6803","nonce":3090454830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6807","nonce":3090454830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":3090454830},{"type":"v1","addr":"172.21.15.114:6805","nonce":3090454830}]},"public_addr":"172.21.15.114:6801/3090454830","cluster_addr":"172.21.15.114:6803/3090454830","heartbeat_back_addr":"172.21.15.114:6807/3090454830","heartbeat_front_addr":"172.21.15.114:6805/3090454830","state":["exists","up"]},{"osd":3,"uuid":"5052332b-1c7e-42e7-9ae6-4751b86b2986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6811","nonce":3120222680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6813","nonce":3120222680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6817","nonce":3120222680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":3120222680},{"type":"v1","addr":"172.21.15.63:6815","nonce":3120222680}]},"public_addr":"172.21.15.63:6811/3120222680","cluster_addr":"172.21.15.63:6813/3120222680","heartbeat_back_addr":"172.21.15.63:6817/3120222680","heartbeat_front_addr":"172.21.15.63:6815/3120222680","state":["exists","up"]},{"osd":4,"uuid":"0da5a992-e540-4065-a95a-2fe0eb881da5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6809","nonce":2157462947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6811","nonce":2157462947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6815","nonce":2157462947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":2157462947},{"type":"v1","addr":"172.21.15.70:6813","nonce":2157462947}]},"public_addr":"172.21.15.70:6809/2157462947","cluster_addr":"172.21.15.70:6811/2157462947","heartbeat_back_addr":"172.21.15.70:6815/2157462947","heartbeat_front_addr":"172.21.15.70:6813/2157462947","state":["exists","up"]},{"osd":5,"uuid":"8832dbef-9e3e-4abf-8ac4-d279ca3af434","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6809","nonce":4152604571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6811","nonce":4152604571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6815","nonce":4152604571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":4152604571},{"type":"v1","addr":"172.21.15.114:6813","nonce":4152604571}]},"public_addr":"172.21.15.114:6809/4152604571","cluster_addr":"172.21.15.114:6811/4152604571","heartbeat_back_addr":"172.21.15.114:6815/4152604571","heartbeat_front_addr":"172.21.15.114:6813/4152604571","state":["exists","up"]},{"osd":6,"uuid":"ca98c42c-8187-4097-a802-137756444990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6819","nonce":3710200907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6821","nonce":3710200907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6825","nonce":3710200907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3710200907},{"type":"v1","addr":"172.21.15.63:6823","nonce":3710200907}]},"public_addr":"172.21.15.63:6819/3710200907","cluster_addr":"172.21.15.63:6821/3710200907","heartbeat_back_addr":"172.21.15.63:6825/3710200907","heartbeat_front_addr":"172.21.15.63:6823/3710200907","state":["exists","up"]},{"osd":7,"uuid":"164885fd-3ba5-4de5-9224-0de701470a27","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6817","nonce":568332364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6819","nonce":568332364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6823","nonce":568332364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":568332364},{"type":"v1","addr":"172.21.15.70:6821","nonce":568332364}]},"public_addr":"172.21.15.70:6817/568332364","cluster_addr":"172.21.15.70:6819/568332364","heartbeat_back_addr":"172.21.15.70:6823/568332364","heartbeat_front_addr":"172.21.15.70:6821/568332364","state":["exists","up"]},{"osd":8,"uuid":"820c8864-3156-4c9f-b583-0360fa2455eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6817","nonce":1230878989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6819","nonce":1230878989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6823","nonce":1230878989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1230878989},{"type":"v1","addr":"172.21.15.114:6821","nonce":1230878989}]},"public_addr":"172.21.15.114:6817/1230878989","cluster_addr":"172.21.15.114:6819/1230878989","heartbeat_back_addr":"172.21.15.114:6823/1230878989","heartbeat_front_addr":"172.21.15.114:6821/1230878989","state":["exists","up"]},{"osd":9,"uuid":"5f01be5b-f17a-4fc6-a8cb-10630b362163","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6827","nonce":2032604816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6829","nonce":2032604816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6833","nonce":2032604816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":2032604816},{"type":"v1","addr":"172.21.15.63:6831","nonce":2032604816}]},"public_addr":"172.21.15.63:6827/2032604816","cluster_addr":"172.21.15.63:6829/2032604816","heartbeat_back_addr":"172.21.15.63:6833/2032604816","heartbeat_front_addr":"172.21.15.63:6831/2032604816","state":["exists","up"]},{"osd":10,"uuid":"067a5560-1dc0-45a8-8a1e-82061039239e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6825","nonce":3803988922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6827","nonce":3803988922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6831","nonce":3803988922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":3803988922},{"type":"v1","addr":"172.21.15.70:6829","nonce":3803988922}]},"public_addr":"172.21.15.70:6825/3803988922","cluster_addr":"172.21.15.70:6827/3803988922","heartbeat_back_addr":"172.21.15.70:6831/3803988922","heartbeat_front_addr":"172.21.15.70:6829/3803988922","state":["exists","up"]},{"osd":11,"uuid":"10cb8d84-790d-42b3-8f29-3771c44434e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":84,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6825","nonce":3297229799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6826","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6827","nonce":3297229799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6830","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6831","nonce":3297229799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6828","nonce":3297229799},{"type":"v1","addr":"172.21.15.114:6829","nonce":3297229799}]},"public_addr":"172.21.15.114:6825/3297229799","cluster_addr":"172.21.15.114:6827/3297229799","heartbeat_back_addr":"172.21.15.114:6831/3297229799","heartbeat_front_addr":"172.21.15.114:6829/3297229799","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:35:52.667165+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:19.175325+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:36:43.974419+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:19.436313+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:37:45.736657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:10.268066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:38:46.322477+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:13.182038+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:39:38.224879+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:14.506594+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:40:41.508045+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T18:41:07.214115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":6}]},{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4},{"from":7,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":7}]},{"pgid":"3.1a","mappings":[{"from":5,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.29","mappings":[{"from":3,"to":9}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/3012688827":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/3560456479":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/2327478809":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/3648036576":"2024-04-26T18:33:52.960272+0000","172.21.15.63:0/1446926443":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/3403991527":"2024-04-26T18:33:52.960272+0000","172.21.15.63:6801/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:6801/3659595029":"2024-04-26T18:46:32.881015+0000","172.21.15.63:6800/2210114679":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/4120416918":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/1747359887":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/3611634789":"2024-04-26T18:46:32.881015+0000","172.21.15.63:0/1119313138":"2024-04-26T18:33:16.955897+0000","172.21.15.63:0/434371372":"2024-04-26T18:33:52.960272+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":24,"length":1},{"begin":36,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":24,"length":1},{"begin":36,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:07:10.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 msgr2=0x7f59040a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:10.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 0x7f59040a6270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5908009680 tx=0x7f5908009260 comp rx=0 tx=0).stop 2024-04-25T19:07:10.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 msgr2=0x7f592019f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:10.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f591000e9f0 tx=0x7f591000ed00 comp rx=0 tx=0).stop 2024-04-25T19:07:10.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 shutdown_connections 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f59040a3db0 0x7f59040a6270 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f5920107970 0x7f59201a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f592006ac90 0x7f592019f720 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 --2- 172.21.15.63:0/2407513602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5920105480 0x7f592019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.413+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 >> 172.21.15.63:0/2407513602 conn(0x7f59200788f0 msgr2=0x7f5920104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:10.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.414+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 shutdown_connections 2024-04-25T19:07:10.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:10.414+0000 7f5925e00700 1 -- 172.21.15.63:0/2407513602 wait complete. 2024-04-25T19:07:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:10 smithi070 ceph-mon[32820]: pgmap v652: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:10.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:10 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2407513602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:07:11.066 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:10 smithi063 ceph-mon[28006]: pgmap v652: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:11.067 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:10 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2407513602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:07:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:10 smithi114 ceph-mon[32988]: pgmap v652: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:11.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:10 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2407513602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T19:07:11.161 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs fail cephfs 2024-04-25T19:07:11.630 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:12.946 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:12 smithi063 ceph-mon[28006]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:12.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:12 smithi070 ceph-mon[32820]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:13.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:12 smithi114 ceph-mon[32988]: pgmap v653: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:13.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 -- 172.21.15.63:0/3387713504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5624109c80 msgr2=0x7f562410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:13.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 --2- 172.21.15.63:0/3387713504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5624109c80 0x7f562410dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f56100099c0 tx=0x7f5610009cd0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 -- 172.21.15.63:0/3387713504 shutdown_connections 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 --2- 172.21.15.63:0/3387713504 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 --2- 172.21.15.63:0/3387713504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f562410e3b0 0x7f5624115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 --2- 172.21.15.63:0/3387713504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5624109c80 0x7f562410dcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 -- 172.21.15.63:0/3387713504 >> 172.21.15.63:0/3387713504 conn(0x7f5624076210 msgr2=0x7f5624107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 -- 172.21.15.63:0/3387713504 shutdown_connections 2024-04-25T19:07:13.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.795+0000 7f562c200700 1 -- 172.21.15.63:0/3387713504 wait complete. 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 Processor -- start 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 -- start start 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 0x7f56241a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 0x7f562419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.797+0000 7f562c200700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5624119270 con 0x7f562410e3b0 2024-04-25T19:07:13.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f562c200700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f56241190f0 con 0x7f5624109c80 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f562c200700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f56241193f0 con 0x7f56241092d0 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f562a200700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 0x7f562419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:38418/0 (socket says 172.21.15.63:38418) 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 -- 172.21.15.63:0/2396399502 learned_addr learned my addr 172.21.15.63:0/2396399502 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5628e00700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 0x7f56241a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 -- 172.21.15.63:0/2396399502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 msgr2=0x7f56241a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 0x7f56241a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 -- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 msgr2=0x7f562419f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 0x7f562419f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5629800700 1 -- 172.21.15.63:0/2396399502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562419fd30 con 0x7f56241092d0 2024-04-25T19:07:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f562a200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 0x7f562419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:07:13.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.798+0000 7f5628e00700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 0x7f56241a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f5629800700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f561800ef10 tx=0x7f561800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561800c9c0 con 0x7f56241092d0 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5618019070 con 0x7f56241092d0 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562419ffc0 con 0x7f56241092d0 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618014480 con 0x7f56241092d0 2024-04-25T19:07:13.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.799+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56241ac2e0 con 0x7f56241092d0 2024-04-25T19:07:13.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.801+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5624069730 con 0x7f56241092d0 2024-04-25T19:07:13.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.803+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f5618012050 con 0x7f56241092d0 2024-04-25T19:07:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.804+0000 7f5622000700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 0x7f560c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.805+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8631+0+0 (secure 0 0 0) 0x7f56180d0790 con 0x7f56241092d0 2024-04-25T19:07:13.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.805+0000 7f5628e00700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 0x7f560c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:13.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.806+0000 7f5628e00700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 0x7f560c0a6270 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5610000c00 tx=0x7f5610005fb0 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:13.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:13.809+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5618010040 con 0x7f56241092d0 2024-04-25T19:07:14.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.098+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f562402df80 con 0x7f56241092d0 2024-04-25T19:07:14.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.106+0000 7f5622000700 1 -- 172.21.15.63:0/2396399502 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v63) ==== 161+0+0 (secure 0 0 0) 0x7f561809cdc0 con 0x7f56241092d0 2024-04-25T19:07:14.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 msgr2=0x7f560c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:14.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 0x7f560c0a6270 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5610000c00 tx=0x7f5610005fb0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 msgr2=0x7f56241a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f561800ef10 tx=0x7f561800c3a0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 shutdown_connections 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f560c0a3db0 0x7f560c0a6270 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f56241092d0 0x7f56241a5410 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f562410e3b0 0x7f562419f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 --2- 172.21.15.63:0/2396399502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5624109c80 0x7f56241a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.109+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 >> 172.21.15.63:0/2396399502 conn(0x7f5624076210 msgr2=0x7f56240768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.110+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 shutdown_connections 2024-04-25T19:07:14.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:14.110+0000 7f562c200700 1 -- 172.21.15.63:0/2396399502 wait complete. 2024-04-25T19:07:14.113 INFO:teuthology.orchestra.run.smithi063.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:07:14.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a[28002]: 2024-04-25T19:07:14.099+0000 7f9244c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:07:14.454 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.454 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mds[62903]: -- 172.21.15.70:0/3662069791 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562f5bee5a40 con 0x562f5beb8800 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: starting mds.e at 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mds[63221]: -- 172.21.15.70:0/777099448 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b31af3ba40 con 0x55b31af0ec00 2024-04-25T19:07:14.455 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: starting mds.h at 2024-04-25T19:07:14.456 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.456 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.456 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mds[63530]: -- 172.21.15.70:0/867638566 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x558fad6ca4e0 con 0x558fad6a0800 2024-04-25T19:07:14.456 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mds[63530]: -- 172.21.15.70:0/867638566 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558fad6cda40 con 0x558fad6a0800 2024-04-25T19:07:14.456 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: starting mds.b at 2024-04-25T19:07:14.569 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.570 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.570 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mds[63106]: -- 172.21.15.114:0/70595935 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dc1fcada40 con 0x55dc1fc80c00 2024-04-25T19:07:14.570 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: starting mds.f at 2024-04-25T19:07:14.570 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.570 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mds[63414]: -- 172.21.15.114:0/4195016045 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55750812c4e0 con 0x557508102c00 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mds[63414]: -- 172.21.15.114:0/4195016045 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55750812fa40 con 0x557508102c00 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: starting mds.c at 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: ignoring --setuser ceph since I am not root 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: ignoring --setgroup ceph since I am not root 2024-04-25T19:07:14.571 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mds[63721]: -- 172.21.15.114:0/513027357 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c931e0c4e0 con 0x55c931de2800 2024-04-25T19:07:14.572 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mds[63721]: -- 172.21.15.114:0/513027357 <== mon.1 v2:172.21.15.114:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c931e0fa40 con 0x55c931de2800 2024-04-25T19:07:14.572 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: starting mds.i at 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2396399502' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:07:14.774 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:07:14.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:07:14.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T19:07:14.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T19:07:14.775 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:14 smithi063 ceph-mon[28006]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T19:07:14.836 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T19:07:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2396399502' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:07:14.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:07:14.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:07:14.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T19:07:14.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T19:07:14.955 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:14 smithi070 ceph-mon[32820]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T19:07:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: pgmap v654: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 88 KiB/s wr, 30 op/s 2024-04-25T19:07:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2396399502' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T19:07:15.067 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T19:07:15.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:07:15.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T19:07:15.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: osdmap e129: 12 total, 12 up, 12 in 2024-04-25T19:07:15.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-25T19:07:15.068 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:14 smithi114 ceph-mon[32988]: osdmap e130: 12 total, 12 up, 12 in 2024-04-25T19:07:15.302 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6834/3209167813,v1:172.21.15.70:6837/3209167813] up:boot 2024-04-25T19:07:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6835/3801306683,v1:172.21.15.70:6838/3801306683] up:boot 2024-04-25T19:07:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.70:6836/1749214722,v1:172.21.15.70:6839/1749214722] up:boot 2024-04-25T19:07:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6834/1342705591,v1:172.21.15.114:6837/1342705591] up:boot 2024-04-25T19:07:16.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6835/2637743739,v1:172.21.15.114:6838/2637743739] up:boot 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: mds.? [v2:172.21.15.114:6836/930521875,v1:172.21.15.114:6839/930521875] up:boot 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:07:16.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:16 smithi063 ceph-mon[28006]: pgmap v657: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 11 op/s 2024-04-25T19:07:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6834/3209167813,v1:172.21.15.70:6837/3209167813] up:boot 2024-04-25T19:07:16.454 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6835/3801306683,v1:172.21.15.70:6838/3801306683] up:boot 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.70:6836/1749214722,v1:172.21.15.70:6839/1749214722] up:boot 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6834/1342705591,v1:172.21.15.114:6837/1342705591] up:boot 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6835/2637743739,v1:172.21.15.114:6838/2637743739] up:boot 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: mds.? [v2:172.21.15.114:6836/930521875,v1:172.21.15.114:6839/930521875] up:boot 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T19:07:16.455 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:07:16.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:07:16.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:07:16.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:07:16.456 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:16 smithi070 ceph-mon[32820]: pgmap v657: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 11 op/s 2024-04-25T19:07:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6834/3209167813,v1:172.21.15.70:6837/3209167813] up:boot 2024-04-25T19:07:16.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6835/3801306683,v1:172.21.15.70:6838/3801306683] up:boot 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.70:6836/1749214722,v1:172.21.15.70:6839/1749214722] up:boot 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6834/1342705591,v1:172.21.15.114:6837/1342705591] up:boot 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6835/2637743739,v1:172.21.15.114:6838/2637743739] up:boot 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: mds.? [v2:172.21.15.114:6836/930521875,v1:172.21.15.114:6839/930521875] up:boot 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T19:07:16.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T19:07:16.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T19:07:16.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T19:07:16.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T19:07:16.569 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:16 smithi114 ceph-mon[32988]: pgmap v657: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 11 op/s 2024-04-25T19:07:17.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 -- 172.21.15.63:0/1090256860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac109c80 msgr2=0x7efeac10dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 --2- 172.21.15.63:0/1090256860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac109c80 0x7efeac10dcd0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7efe980098f0 tx=0x7efe98009c00 comp rx=0 tx=0).stop 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 -- 172.21.15.63:0/1090256860 shutdown_connections 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 --2- 172.21.15.63:0/1090256860 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 0x7efeac115f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 --2- 172.21.15.63:0/1090256860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac1092d0 0x7efeac1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 --2- 172.21.15.63:0/1090256860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac109c80 0x7efeac10dcd0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 -- 172.21.15.63:0/1090256860 >> 172.21.15.63:0/1090256860 conn(0x7efeac076210 msgr2=0x7efeac107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:17.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 -- 172.21.15.63:0/1090256860 shutdown_connections 2024-04-25T19:07:17.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.216+0000 7efeb1000700 1 -- 172.21.15.63:0/1090256860 wait complete. 2024-04-25T19:07:17.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.218+0000 7efeb1000700 1 Processor -- start 2024-04-25T19:07:17.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.218+0000 7efeb1000700 1 -- start start 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.218+0000 7efeb1000700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeb1000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac109c80 0x7efeac19fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeb1000700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeb1000700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efeac119220 con 0x7efeac109c80 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeb1000700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1190a0 con 0x7efeac1092d0 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeb1000700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7efeac1193a0 con 0x7efeac10e3b0 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:17.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9400700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac109c80 0x7efeac19fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.63:33454/0 (socket says 172.21.15.63:33454) 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 -- 172.21.15.63:0/94905399 learned_addr learned my addr 172.21.15.63:0/94905399 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efeaa800700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 -- 172.21.15.63:0/94905399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 msgr2=0x7efeac1a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 -- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac109c80 msgr2=0x7efeac19fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac109c80 0x7efeac19fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.219+0000 7efea9e00700 1 -- 172.21.15.63:0/94905399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac1a4710 con 0x7efeac1092d0 2024-04-25T19:07:17.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efea9e00700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9a0 tx=0x7efe9c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:17.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00ca10 con 0x7efeac1092d0 2024-04-25T19:07:17.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac1a4a00 con 0x7efeac1092d0 2024-04-25T19:07:17.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe9c00cbb0 con 0x7efeac1092d0 2024-04-25T19:07:17.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c018680 con 0x7efeac1092d0 2024-04-25T19:07:17.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.220+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac1ac280 con 0x7efeac1092d0 2024-04-25T19:07:17.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.221+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeac069730 con 0x7efeac1092d0 2024-04-25T19:07:17.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.224+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7efe9c010430 con 0x7efeac1092d0 2024-04-25T19:07:17.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.225+0000 7efea2a00700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 0x7efe900a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:17.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.225+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8845+0+0 (secure 0 0 0) 0x7efe9c0d0d80 con 0x7efeac1092d0 2024-04-25T19:07:17.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.226+0000 7efea9400700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 0x7efe900a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:17.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.226+0000 7efea9400700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 0x7efe900a62f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe98009810 tx=0x7efe98019040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:17.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.227+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe9c09d280 con 0x7efeac1092d0 2024-04-25T19:07:17.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:17.509+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7efeac02df80 con 0x7efeac1092d0 2024-04-25T19:07:18.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.502+0000 7efea2a00700 1 -- 172.21.15.63:0/94905399 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v65) ==== 104+0+0 (secure 0 0 0) 0x7efe9c09cb80 con 0x7efeac1092d0 2024-04-25T19:07:18.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.506+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 msgr2=0x7efe900a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:18.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.506+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 0x7efe900a62f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7efe98009810 tx=0x7efe98019040 comp rx=0 tx=0).stop 2024-04-25T19:07:18.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.506+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 msgr2=0x7efeac19f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:18.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.506+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00e9a0 tx=0x7efe9c00ecb0 comp rx=0 tx=0).stop 2024-04-25T19:07:18.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 shutdown_connections 2024-04-25T19:07:18.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7efe900a3e30 0x7efe900a62f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:18.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7efeac10e3b0 0x7efeac1a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:18.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efeac1092d0 0x7efeac19f750 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:18.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 --2- 172.21.15.63:0/94905399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efeac109c80 0x7efeac19fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:18.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 >> 172.21.15.63:0/94905399 conn(0x7efeac076210 msgr2=0x7efeac077dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:18.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 shutdown_connections 2024-04-25T19:07:18.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:18.507+0000 7efeb1000700 1 -- 172.21.15.63:0/94905399 wait complete. 2024-04-25T19:07:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:18 smithi114 ceph-mon[32988]: pgmap v658: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 4 op/s 2024-04-25T19:07:18.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:18 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/94905399' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:18.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:18 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:18 smithi063 ceph-mon[28006]: pgmap v658: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 4 op/s 2024-04-25T19:07:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:18 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/94905399' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:18.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:18 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:18 smithi070 ceph-mon[32820]: pgmap v658: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 4 op/s 2024-04-25T19:07:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:18 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/94905399' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:18.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:18 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T19:07:19.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:07:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:19 smithi114 ceph-mon[32988]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:07:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:19 smithi114 ceph-mon[32988]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:07:19.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:19 smithi114 ceph-mon[32988]: Cluster is now healthy 2024-04-25T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:19 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:19 smithi114 ceph-mon[32988]: fsmap 12 up:standby 2024-04-25T19:07:19.822 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:19.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:19 smithi063 ceph-mon[28006]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:07:19.877 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:19 smithi063 ceph-mon[28006]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:07:19.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:19 smithi063 ceph-mon[28006]: Cluster is now healthy 2024-04-25T19:07:19.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:19 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:07:19.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:19 smithi063 ceph-mon[28006]: fsmap 12 up:standby 2024-04-25T19:07:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:19 smithi070 ceph-mon[32820]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T19:07:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:19 smithi070 ceph-mon[32820]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T19:07:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:19 smithi070 ceph-mon[32820]: Cluster is now healthy 2024-04-25T19:07:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:19 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T19:07:19.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:19 smithi070 ceph-mon[32820]: fsmap 12 up:standby 2024-04-25T19:07:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:20 smithi114 ceph-mon[32988]: pgmap v659: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:20 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:20 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:20.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:20 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:20 smithi063 ceph-mon[28006]: pgmap v659: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:20 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:20 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:20.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:20 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:20 smithi070 ceph-mon[32820]: pgmap v659: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:20 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T19:07:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:20 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:20.954 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:20 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:21.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.802+0000 7f300ee00700 1 -- 172.21.15.63:0/1727648704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008107990 msgr2=0x7f3008107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.802+0000 7f300ee00700 1 --2- 172.21.15.63:0/1727648704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008107990 0x7f3008107d70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80098f0 tx=0x7f2ff8009c00 comp rx=0 tx=0).stop 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 -- 172.21.15.63:0/1727648704 shutdown_connections 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 --2- 172.21.15.63:0/1727648704 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008107990 0x7f3008107d70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 --2- 172.21.15.63:0/1727648704 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008102300 0x7f300810e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 --2- 172.21.15.63:0/1727648704 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 0x7f3008101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:21.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 -- 172.21.15.63:0/1727648704 >> 172.21.15.63:0/1727648704 conn(0x7f30080788f0 msgr2=0x7f3008078d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:21.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 -- 172.21.15.63:0/1727648704 shutdown_connections 2024-04-25T19:07:21.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.803+0000 7f300ee00700 1 -- 172.21.15.63:0/1727648704 wait complete. 2024-04-25T19:07:21.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.804+0000 7f300ee00700 1 Processor -- start 2024-04-25T19:07:21.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.804+0000 7f300ee00700 1 -- start start 2024-04-25T19:07:21.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 0x7f30081034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008107990 0x7f3008103f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3008111b30 con 0x7f3008107990 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30081119b0 con 0x7f3008101940 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ee00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f3008111cb0 con 0x7f3008102300 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f3007400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f3007400700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.114:3300/0 says I am v2:172.21.15.63:35222/0 (socket says 172.21.15.63:35222) 2024-04-25T19:07:21.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f3007400700 1 -- 172.21.15.63:0/197825466 learned_addr learned my addr 172.21.15.63:0/197825466 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f3007e00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 0x7f30081034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.805+0000 7f300ce00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008107990 0x7f3008103f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 -- 172.21.15.63:0/197825466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 msgr2=0x7f30081034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 0x7f30081034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 -- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008107990 msgr2=0x7f3008103f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008107990 0x7f3008103f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 -- 172.21.15.63:0/197825466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30081a7690 con 0x7f3008102300 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3007400700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400ec10 tx=0x7f2ff400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:21.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400cd40 con 0x7f3008102300 2024-04-25T19:07:21.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff40044d0 con 0x7f3008102300 2024-04-25T19:07:21.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4010690 con 0x7f3008102300 2024-04-25T19:07:21.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.806+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30081a7980 con 0x7f3008102300 2024-04-25T19:07:21.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.807+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30081a7e70 con 0x7f3008102300 2024-04-25T19:07:21.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.808+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3008069730 con 0x7f3008102300 2024-04-25T19:07:21.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.810+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f2ff4003e50 con 0x7f3008102300 2024-04-25T19:07:21.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.811+0000 7f3004c00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 0x7f2fec0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:21.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.811+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 8845+0+0 (secure 0 0 0) 0x7f2ff40d0c70 con 0x7f3008102300 2024-04-25T19:07:21.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.811+0000 7f3007e00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 0x7f2fec0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:21.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.812+0000 7f3007e00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 0x7f2fec0a62f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80098c0 tx=0x7f2ff8019040 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:21.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:21.816+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ff409d170 con 0x7f3008102300 2024-04-25T19:07:22.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:22.071+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3008104a20 con 0x7f3008102300 2024-04-25T19:07:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: pgmap v660: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/197825466' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:22.630 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:22 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: pgmap v660: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/197825466' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:22.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:22 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: pgmap v660: 97 pgs: 97 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 16 KiB/s wr, 4 op/s 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/197825466' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:22.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:22 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:23.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.029+0000 7f3004c00700 1 -- 172.21.15.63:0/197825466 <== mon.1 v2:172.21.15.114:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v131) ==== 181+0+0 (secure 0 0 0) 0x7f2ff409ca70 con 0x7f3008102300 2024-04-25T19:07:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 msgr2=0x7f2fec0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 0x7f2fec0a62f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80098c0 tx=0x7f2ff8019040 comp rx=0 tx=0).stop 2024-04-25T19:07:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 msgr2=0x7f3008103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400ec10 tx=0x7f2ff400ef20 comp rx=0 tx=0).stop 2024-04-25T19:07:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 shutdown_connections 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f2fec0a3e30 0x7f2fec0a62f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3008107990 0x7f3008103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3008101940 0x7f30081034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 --2- 172.21.15.63:0/197825466 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f3008102300 0x7f3008103a30 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400ec10 tx=0x7f2ff400ef20 comp rx=0 tx=0).stop 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.034+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 >> 172.21.15.63:0/197825466 conn(0x7f30080788f0 msgr2=0x7f300806a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:23.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.035+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 shutdown_connections 2024-04-25T19:07:23.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:23.035+0000 7f300ee00700 1 -- 172.21.15.63:0/197825466 wait complete. 2024-04-25T19:07:23.044 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'cephfs_metadata' removed 2024-04-25T19:07:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T19:07:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T19:07:23.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:23.629 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:23 smithi063 ceph-mon[28006]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T19:07:23.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T19:07:23.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:23.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:23 smithi070 ceph-mon[32820]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T19:07:23.783 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f shell --fsid e90c993c-0331-11ef-bc93-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:07:23.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-04-25T19:07:23.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi070 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi114 to 176.2M 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi114 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-25T19:07:23.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:23.819 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:23 smithi114 ceph-mon[32988]: osdmap e131: 12 total, 12 up, 12 in 2024-04-25T19:07:24.247 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T19:07:24.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:24 smithi063 ceph-mon[28006]: pgmap v662: 65 pgs: 65 active+clean; 315 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:24.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:24 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:24.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:24 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:24.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:24 smithi070 ceph-mon[32820]: pgmap v662: 65 pgs: 65 active+clean; 315 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:24.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:24 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:24.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:24 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:24 smithi114 ceph-mon[32988]: pgmap v662: 65 pgs: 65 active+clean; 315 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:24 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:24.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:24 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:25.429 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:25.430 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:25 smithi063 ceph-mon[28006]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:25.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.567 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:25.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:25.568 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:25 smithi114 ceph-mon[32988]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T19:07:25.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T19:07:25.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T19:07:25.705 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:25 smithi070 ceph-mon[32820]: from='mgr.25069 172.21.15.63:0/3581718003' entity='mgr.x' 2024-04-25T19:07:26.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 -- 172.21.15.63:0/1365119997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 msgr2=0x7f9c7c108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 --2- 172.21.15.63:0/1365119997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c108e00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0098f0 tx=0x7f9c6c009c00 comp rx=0 tx=0).stop 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 -- 172.21.15.63:0/1365119997 shutdown_connections 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 --2- 172.21.15.63:0/1365119997 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c108e00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 --2- 172.21.15.63:0/1365119997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 --2- 172.21.15.63:0/1365119997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c06b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.249+0000 7f9c81a00700 1 -- 172.21.15.63:0/1365119997 >> 172.21.15.63:0/1365119997 conn(0x7f9c7c0788b0 msgr2=0x7f9c7c078cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.250+0000 7f9c81a00700 1 -- 172.21.15.63:0/1365119997 shutdown_connections 2024-04-25T19:07:26.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.250+0000 7f9c81a00700 1 -- 172.21.15.63:0/1365119997 wait complete. 2024-04-25T19:07:26.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 Processor -- start 2024-04-25T19:07:26.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 -- start start 2024-04-25T19:07:26.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 --2- >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7c112c10 con 0x7f9c7c06b980 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7c112a90 con 0x7f9c7c06afe0 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.251+0000 7f9c81a00700 1 -- --> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7c112d90 con 0x7f9c7c108a20 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7a000700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51826/0 (socket says 172.21.15.63:51826) 2024-04-25T19:07:26.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:26.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7a000700 1 -- 172.21.15.63:0/2888792014 learned_addr learned my addr 172.21.15.63:0/2888792014 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T19:07:26.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7b400700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 -- 172.21.15.63:0/2888792014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 msgr2=0x7f9c7c19fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c19fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 -- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 msgr2=0x7f9c7c19b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7aa00700 1 -- 172.21.15.63:0/2888792014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7c1a0460 con 0x7f9c7c06afe0 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.252+0000 7f9c7a000700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c7aa00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c19b450 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c010 tx=0x7f9c6c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:26.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0050c0 con 0x7f9c7c06afe0 2024-04-25T19:07:26.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c6c00b560 con 0x7f9c7c06afe0 2024-04-25T19:07:26.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c7c1a06f0 con 0x7f9c7c06afe0 2024-04-25T19:07:26.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c003a00 con 0x7f9c7c06afe0 2024-04-25T19:07:26.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.253+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c7c1a7fd0 con 0x7f9c7c06afe0 2024-04-25T19:07:26.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.254+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c7c104160 con 0x7f9c7c06afe0 2024-04-25T19:07:26.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.256+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 26) ==== 138777+0+0 (secure 0 0 0) 0x7f9c6c00b710 con 0x7f9c7c06afe0 2024-04-25T19:07:26.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.258+0000 7f9c73400700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 0x7f9c600a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:07:26.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.258+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 8231+0+0 (secure 0 0 0) 0x7f9c6c0d1760 con 0x7f9c7c06afe0 2024-04-25T19:07:26.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.258+0000 7f9c7a000700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 0x7f9c600a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:07:26.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.259+0000 7f9c7a000700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 0x7f9c600a6270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c078d40 tx=0x7f9c68009260 comp rx=0 tx=0).ready entity=mgr.25069 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:07:26.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.260+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c6c013030 con 0x7f9c7c06afe0 2024-04-25T19:07:26.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:26.507+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9c7c02df80 con 0x7f9c7c06afe0 2024-04-25T19:07:26.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:26 smithi063 ceph-mon[28006]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T19:07:26.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:26 smithi063 ceph-mon[28006]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T19:07:26.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:26 smithi063 ceph-mon[28006]: pgmap v663: 65 pgs: 65 active+clean; 315 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:26 smithi070 ceph-mon[32820]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T19:07:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:26 smithi070 ceph-mon[32820]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T19:07:26.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:26 smithi070 ceph-mon[32820]: pgmap v663: 65 pgs: 65 active+clean; 315 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:26 smithi114 ceph-mon[32988]: Adjusting osd_memory_target on smithi063 to 176.4M 2024-04-25T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:26 smithi114 ceph-mon[32988]: Unable to set osd_memory_target on smithi063 to 185070592: error parsing value: Value '185070592' is below minimum 939524096 2024-04-25T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:26 smithi114 ceph-mon[32988]: pgmap v663: 65 pgs: 65 active+clean; 315 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:27.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.382+0000 7f9c73400700 1 -- 172.21.15.63:0/2888792014 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v132) ==== 169+0+0 (secure 0 0 0) 0x7f9c6c021380 con 0x7f9c7c06afe0 2024-04-25T19:07:27.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.391+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 msgr2=0x7f9c600a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:27.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.391+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 0x7f9c600a6270 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c078d40 tx=0x7f9c68009260 comp rx=0 tx=0).stop 2024-04-25T19:07:27.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.391+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 msgr2=0x7f9c7c19b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:07:27.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.391+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c19b450 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c00c010 tx=0x7f9c6c00fa00 comp rx=0 tx=0).stop 2024-04-25T19:07:27.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 shutdown_connections 2024-04-25T19:07:27.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:6800/2505142895,v1:172.21.15.63:6801/2505142895] conn(0x7f9c600a3db0 0x7f9c600a6270 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:27.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] conn(0x7f9c7c108a20 0x7f9c7c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:27.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9c7c06afe0 0x7f9c7c19b450 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:27.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 --2- 172.21.15.63:0/2888792014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c7c06b980 0x7f9c7c19b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:07:27.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.392+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 >> 172.21.15.63:0/2888792014 conn(0x7f9c7c0788b0 msgr2=0x7f9c7c10dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:07:27.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.394+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 shutdown_connections 2024-04-25T19:07:27.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T19:07:27.394+0000 7f9c81a00700 1 -- 172.21.15.63:0/2888792014 wait complete. 2024-04-25T19:07:27.400 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'cephfs_data' removed 2024-04-25T19:07:27.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:27 smithi063 ceph-mon[28006]: from='client.? 172.21.15.63:0/2888792014' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:27.628 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:27 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:27.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:27 smithi070 ceph-mon[32820]: from='client.? 172.21.15.63:0/2888792014' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:27.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:27 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:27.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:27 smithi114 ceph-mon[32988]: from='client.? 172.21.15.63:0/2888792014' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:27.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:27 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T19:07:28.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:07:28.374 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:07:28.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:28 smithi063 ceph-mon[28006]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:28.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:28 smithi063 ceph-mon[28006]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T19:07:28.391 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:28 smithi063 ceph-mon[28006]: pgmap v665: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:28.402 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:07:28.429 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T19:07:28.430 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:07:28.472 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:07:28.489 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T19:07:28.504 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T19:07:28.504 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T19:07:28.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a 2024-04-25T19:07:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:28 smithi070 ceph-mon[32820]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:28 smithi070 ceph-mon[32820]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T19:07:28.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:28 smithi070 ceph-mon[32820]: pgmap v665: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:28 smithi114 ceph-mon[32988]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T19:07:28.817 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:28 smithi114 ceph-mon[32988]: osdmap e132: 12 total, 12 up, 12 in 2024-04-25T19:07:28.818 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:28 smithi114 ceph-mon[32988]: pgmap v665: 1 pgs: 1 active+clean; 577 KiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T19:07:28.878 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:28 smithi063 systemd[1]: Stopping Ceph mon.a for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:29.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a[28002]: 2024-04-25T19:07:29.099+0000 7f924ba00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:29.378 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a[28002]: 2024-04-25T19:07:29.099+0000 7f924ba00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T19:07:29.379 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 podman[108210]: 2024-04-25 19:07:29.213241434 +0000 UTC m=+0.263655644 container died 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-25T19:07:30.001 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 podman[108210]: 2024-04-25 19:07:29.739615853 +0000 UTC m=+0.790030063 container cleanup 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T19:07:30.001 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 bash[108210]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a 2024-04-25T19:07:30.377 INFO:journalctl@ceph.mon.a.smithi063.stdout:Apr 25 19:07:29 smithi063 podman[108224]: 2024-04-25 19:07:29.999151105 +0000 UTC m=+0.765026472 container remove 329dd1f8f9f9e58260f299c4278e2e282122df86c7f89921b70858eaacf18171 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T19:07:30.531 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T19:07:30.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:30.688 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T19:07:30.688 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T19:07:30.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.b 2024-04-25T19:07:30.996 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:30 smithi070 systemd[1]: Stopping Ceph mon.b for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:31.386 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b[32816]: 2024-04-25T19:07:30.993+0000 7f36c96cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:31.387 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b[32816]: 2024-04-25T19:07:30.993+0000 7f36c96cb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T19:07:31.387 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:31 smithi070 podman[72755]: 2024-04-25 19:07:31.014374722 +0000 UTC m=+0.129054030 container died 229d68a239fe78ba4a5f3293037c1d4f0119695a4c72a0c8b43336b8fd22f949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T19:07:31.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:31 smithi070 podman[72755]: 2024-04-25 19:07:31.484934676 +0000 UTC m=+0.599613999 container cleanup 229d68a239fe78ba4a5f3293037c1d4f0119695a4c72a0c8b43336b8fd22f949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-04-25T19:07:31.704 INFO:journalctl@ceph.mon.b.smithi070.stdout:Apr 25 19:07:31 smithi070 bash[72755]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-b 2024-04-25T19:07:31.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T19:07:31.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:31.936 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T19:07:31.937 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T19:07:31.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.c 2024-04-25T19:07:32.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:31 smithi114 systemd[1]: Stopping Ceph mon.c for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:32.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c[32984]: 2024-04-25T19:07:32.320+0000 7f8b60a79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:32.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c[32984]: 2024-04-25T19:07:32.320+0000 7f8b60a79700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T19:07:32.690 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:32 smithi114 podman[72588]: 2024-04-25 19:07:32.40375515 +0000 UTC m=+0.241071342 container died e4318e7915949b903d95ee5b13a72afe465db2b3d81f901fd9e04b7af187c4d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T19:07:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:32 smithi114 podman[72588]: 2024-04-25 19:07:32.830249746 +0000 UTC m=+0.667565937 container cleanup e4318e7915949b903d95ee5b13a72afe465db2b3d81f901fd9e04b7af187c4d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-25T19:07:33.066 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:32 smithi114 bash[72588]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c 2024-04-25T19:07:33.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T19:07:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:33 smithi114 podman[72602]: 2024-04-25 19:07:33.065025838 +0000 UTC m=+0.656451316 container remove e4318e7915949b903d95ee5b13a72afe465db2b3d81f901fd9e04b7af187c4d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T19:07:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:33 smithi114 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-25T19:07:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:33 smithi114 systemd[1]: Stopped Ceph mon.c for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T19:07:33.317 INFO:journalctl@ceph.mon.c.smithi114.stdout:Apr 25 19:07:33 smithi114 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mon.c.service: Consumed 19.182s CPU time. 2024-04-25T19:07:33.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:33.344 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T19:07:33.344 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T19:07:33.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x 2024-04-25T19:07:33.878 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:33 smithi063 systemd[1]: Stopping Ceph mgr.x for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:34.287 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x[28246]: 2024-04-25T19:07:33.909+0000 7ff498800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:34.287 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:34 smithi063 podman[108331]: 2024-04-25 19:07:34.001956677 +0000 UTC m=+0.240572330 container died fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-25T19:07:34.815 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:34 smithi063 podman[108331]: 2024-04-25 19:07:34.561562793 +0000 UTC m=+0.800178483 container cleanup fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T19:07:34.815 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:34 smithi063 bash[108331]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x 2024-04-25T19:07:35.128 INFO:journalctl@ceph.mgr.x.smithi063.stdout:Apr 25 19:07:34 smithi063 podman[108345]: 2024-04-25 19:07:34.813093808 +0000 UTC m=+0.794516082 container remove fa158b16e4c8ec54a47525dc8a5392125776bc5d0bca88c451d0c89ca1b0e897 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:d69f13d827952bb07bb9b7973bca14f97d764d6f, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T19:07:35.424 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T19:07:35.525 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:35.525 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T19:07:35.525 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T19:07:35.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.y 2024-04-25T19:07:35.876 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 19:07:35 smithi070 systemd[1]: Stopping Ceph mgr.y for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:36.157 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 19:07:35 smithi070 podman[72880]: 2024-04-25 19:07:35.875533617 +0000 UTC m=+0.164332126 container died d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:07:36.454 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 19:07:36 smithi070 podman[72880]: 2024-04-25 19:07:36.263254087 +0000 UTC m=+0.552052596 container cleanup d2e1b3ad2a33f05734259288f16d65e9bc4461329d3eb7b53c1bd8db5a1de7d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:07:36.454 INFO:journalctl@ceph.mgr.y.smithi070.stdout:Apr 25 19:07:36 smithi070 bash[72880]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-y 2024-04-25T19:07:36.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T19:07:36.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:36.782 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T19:07:36.782 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T19:07:36.782 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z 2024-04-25T19:07:37.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:36 smithi114 systemd[1]: Stopping Ceph mgr.z for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:37.529 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:37 smithi114 podman[72709]: 2024-04-25 19:07:37.122488814 +0000 UTC m=+0.154365069 container died cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T19:07:37.812 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:37 smithi114 podman[72709]: 2024-04-25 19:07:37.602096187 +0000 UTC m=+0.633972432 container cleanup cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:07:37.812 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:37 smithi114 bash[72709]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z 2024-04-25T19:07:38.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T19:07:38.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:37 smithi114 podman[72723]: 2024-04-25 19:07:37.811460276 +0000 UTC m=+0.684466567 container remove cb781d44ef4d1d93911044067b40eeb0d567f60120faa1dfb6fea4f6d658e6a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:07:38.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:37 smithi114 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-25T19:07:38.067 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:38 smithi114 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-04-25T19:07:38.068 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:38 smithi114 systemd[1]: Stopped Ceph mgr.z for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T19:07:38.068 INFO:journalctl@ceph.mgr.z.smithi114.stdout:Apr 25 19:07:38 smithi114 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mgr.z.service: Consumed 23.109s CPU time. 2024-04-25T19:07:38.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:38.070 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T19:07:38.070 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T19:07:38.071 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.0 2024-04-25T19:07:38.628 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:38 smithi063 systemd[1]: Stopping Ceph osd.0 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:39.128 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0[42795]: 2024-04-25T19:07:38.688+0000 7fd6c0c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:39.129 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0[42795]: 2024-04-25T19:07:38.689+0000 7fd6c0c00700 -1 osd.0 132 *** Got signal Terminated *** 2024-04-25T19:07:39.129 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0[42795]: 2024-04-25T19:07:38.689+0000 7fd6c0c00700 -1 osd.0 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:07:44.733 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:44 smithi063 podman[108450]: 2024-04-25 19:07:44.438556073 +0000 UTC m=+5.904997339 container died 2fa3e6f40e2ae6861454cc1c52431a9748a4370f5ee5228d072b0f2661c243d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T19:07:45.078 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:44 smithi063 podman[108450]: 2024-04-25 19:07:44.84863539 +0000 UTC m=+6.315076693 container cleanup 2fa3e6f40e2ae6861454cc1c52431a9748a4370f5ee5228d072b0f2661c243d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:07:45.078 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:44 smithi063 bash[108450]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0 2024-04-25T19:07:45.378 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:45 smithi063 podman[108472]: 2024-04-25 19:07:45.074980067 +0000 UTC m=+0.621203434 container remove 2fa3e6f40e2ae6861454cc1c52431a9748a4370f5ee5228d072b0f2661c243d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-25T19:07:46.128 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:45 smithi063 podman[108560]: 2024-04-25 19:07:45.748886204 +0000 UTC m=+0.033162295 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:07:46.129 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:45 smithi063 podman[108560]: 2024-04-25 19:07:45.867475685 +0000 UTC m=+0.151751796 container create bd2646a2d404ba5fd5d961d945c76e8c16fec8ea9476af2e94ae52280e475b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T19:07:46.618 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:46 smithi063 podman[108560]: 2024-04-25 19:07:46.129536802 +0000 UTC m=+0.413812956 container init bd2646a2d404ba5fd5d961d945c76e8c16fec8ea9476af2e94ae52280e475b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-25T19:07:46.619 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:46 smithi063 podman[108560]: 2024-04-25 19:07:46.142102477 +0000 UTC m=+0.426378595 container start bd2646a2d404ba5fd5d961d945c76e8c16fec8ea9476af2e94ae52280e475b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:07:46.619 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:46 smithi063 podman[108560]: 2024-04-25 19:07:46.24471941 +0000 UTC m=+0.528995585 container attach bd2646a2d404ba5fd5d961d945c76e8c16fec8ea9476af2e94ae52280e475b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-25T19:07:46.619 INFO:journalctl@ceph.osd.0.smithi063.stdout:Apr 25 19:07:46 smithi063 podman[108560]: 2024-04-25 19:07:46.616742501 +0000 UTC m=+0.901018588 container died bd2646a2d404ba5fd5d961d945c76e8c16fec8ea9476af2e94ae52280e475b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T19:07:47.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T19:07:47.419 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:47.420 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T19:07:47.420 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T19:07:47.420 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.1 2024-04-25T19:07:47.704 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:47 smithi070 systemd[1]: Stopping Ceph osd.1 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:48.204 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1[37434]: 2024-04-25T19:07:47.723+0000 7f456bbb8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:48.204 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1[37434]: 2024-04-25T19:07:47.723+0000 7f456bbb8700 -1 osd.1 132 *** Got signal Terminated *** 2024-04-25T19:07:48.204 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1[37434]: 2024-04-25T19:07:47.723+0000 7f456bbb8700 -1 osd.1 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:07:53.583 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:53 smithi070 podman[73004]: 2024-04-25 19:07:53.265782104 +0000 UTC m=+5.653887025 container died 881380d11f462a347fe8f17ae3c5d326a2292cee25701f742dc107e5c2f9e830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T19:07:53.867 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:53 smithi070 podman[73004]: 2024-04-25 19:07:53.581532582 +0000 UTC m=+5.969637502 container cleanup 881380d11f462a347fe8f17ae3c5d326a2292cee25701f742dc107e5c2f9e830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:07:53.867 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:53 smithi070 bash[73004]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1 2024-04-25T19:07:54.204 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:53 smithi070 podman[73021]: 2024-04-25 19:07:53.866143534 +0000 UTC m=+0.600333409 container remove 881380d11f462a347fe8f17ae3c5d326a2292cee25701f742dc107e5c2f9e830 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:07:54.645 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.250086799 +0000 UTC m=+0.017695057 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:07:54.646 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.350244498 +0000 UTC m=+0.117852760 container create 0678e989f3b15cb2f078c13e27e2087f8b712fb7915ca84c377dd42ff5f0bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T19:07:54.646 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.577675053 +0000 UTC m=+0.345283311 container init 0678e989f3b15cb2f078c13e27e2087f8b712fb7915ca84c377dd42ff5f0bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T19:07:54.646 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.580808447 +0000 UTC m=+0.348416712 container start 0678e989f3b15cb2f078c13e27e2087f8b712fb7915ca84c377dd42ff5f0bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-25T19:07:54.646 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.644168367 +0000 UTC m=+0.411776635 container attach 0678e989f3b15cb2f078c13e27e2087f8b712fb7915ca84c377dd42ff5f0bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T19:07:54.954 INFO:journalctl@ceph.osd.1.smithi070.stdout:Apr 25 19:07:54 smithi070 podman[73109]: 2024-04-25 19:07:54.874792185 +0000 UTC m=+0.642400435 container died 0678e989f3b15cb2f078c13e27e2087f8b712fb7915ca84c377dd42ff5f0bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T19:07:55.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T19:07:55.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:07:55.490 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T19:07:55.490 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T19:07:55.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.2 2024-04-25T19:07:55.807 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:07:55 smithi114 systemd[1]: Stopping Ceph osd.2 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:07:56.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:07:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2[38142]: 2024-04-25T19:07:55.805+0000 7f95a95c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:07:56.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:07:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2[38142]: 2024-04-25T19:07:55.805+0000 7f95a95c4700 -1 osd.2 132 *** Got signal Terminated *** 2024-04-25T19:07:56.067 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:07:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2[38142]: 2024-04-25T19:07:55.805+0000 7f95a95c4700 -1 osd.2 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:01.500 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:01 smithi114 podman[72844]: 2024-04-25 19:08:01.227673102 +0000 UTC m=+5.547417040 container died 34258b0e80acfd0415fabbbaa2f40fff306292122456df9ee7196823f4fafff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:01.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:01 smithi114 podman[72844]: 2024-04-25 19:08:01.597337471 +0000 UTC m=+5.917081390 container cleanup 34258b0e80acfd0415fabbbaa2f40fff306292122456df9ee7196823f4fafff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-25T19:08:01.817 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:01 smithi114 bash[72844]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2 2024-04-25T19:08:02.192 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:01 smithi114 podman[72860]: 2024-04-25 19:08:01.890433272 +0000 UTC m=+0.664981556 container remove 34258b0e80acfd0415fabbbaa2f40fff306292122456df9ee7196823f4fafff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:02.461 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.191365294 +0000 UTC m=+0.098479484 container create b7ee51a7b70dcf589cab4887e9b3d63ff3a5699a497826e764537c3c8b9aa5d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:02.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.110361113 +0000 UTC m=+0.017475332 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:02.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.394183546 +0000 UTC m=+0.301297738 container init b7ee51a7b70dcf589cab4887e9b3d63ff3a5699a497826e764537c3c8b9aa5d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:02.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.397558089 +0000 UTC m=+0.304672280 container start b7ee51a7b70dcf589cab4887e9b3d63ff3a5699a497826e764537c3c8b9aa5d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416) 2024-04-25T19:08:02.462 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.460349806 +0000 UTC m=+0.367464009 container attach b7ee51a7b70dcf589cab4887e9b3d63ff3a5699a497826e764537c3c8b9aa5d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2-deactivate, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:08:03.031 INFO:journalctl@ceph.osd.2.smithi114.stdout:Apr 25 19:08:02 smithi114 podman[72955]: 2024-04-25 19:08:02.729222015 +0000 UTC m=+0.636336224 container died b7ee51a7b70dcf589cab4887e9b3d63ff3a5699a497826e764537c3c8b9aa5d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T19:08:03.326 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T19:08:03.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:03.366 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T19:08:03.367 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T19:08:03.367 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.3 2024-04-25T19:08:03.848 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:03 smithi063 systemd[1]: Stopping Ceph osd.3 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:04.128 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:03 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:03.946+0000 7f8e33800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:04.128 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:03 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:03.946+0000 7f8e33800700 -1 osd.3 132 *** Got signal Terminated *** 2024-04-25T19:08:04.128 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:03 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:03.946+0000 7f8e33800700 -1 osd.3 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:05.378 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:05 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:05.074+0000 7f8e2a000700 -1 osd.3 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:40.362940+0000 front 2024-04-25T19:07:40.363037+0000 (oldest deadline 2024-04-25T19:08:05.063151+0000) 2024-04-25T19:08:05.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:04 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:04.997+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:05.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:05.100+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:05.813 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:05.541+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:05.879 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:05 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:05.499+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:06.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:05.810+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:06.378 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:06 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:06.107+0000 7f8e2a000700 -1 osd.3 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:40.362940+0000 front 2024-04-25T19:07:40.363037+0000 (oldest deadline 2024-04-25T19:08:05.063151+0000) 2024-04-25T19:08:06.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:05 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:05.969+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:06.136+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:06.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:06 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:06.544+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:07.067 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:06.577+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:07.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:06.784+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:07.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:06.805+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:07.378 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:07.141+0000 7f8e2a000700 -1 osd.3 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:40.362940+0000 front 2024-04-25T19:07:40.363037+0000 (oldest deadline 2024-04-25T19:08:05.063151+0000) 2024-04-25T19:08:07.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:06 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:06.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:07.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:07.124+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:07.809 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:07.536+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:07.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:07.594+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:08.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:07.807+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:08.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:07.807+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:08.378 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3[49756]: 2024-04-25T19:08:08.158+0000 7f8e2a000700 -1 osd.3 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:40.362940+0000 front 2024-04-25T19:07:40.363037+0000 (oldest deadline 2024-04-25T19:08:05.063151+0000) 2024-04-25T19:08:08.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:07 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:07.990+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:08.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:08.083+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:08.801 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:08.535+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:08.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:08.598+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:09.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:08.835+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:09.068 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:08.799+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:09.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:08 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:08.990+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:09.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:09 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:09.038+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:09.796 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:09.491+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:09.878 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:09 smithi063 podman[108664]: 2024-04-25 19:08:09.653163849 +0000 UTC m=+5.819221571 container died 490cf821db8fefb2c97431142ff1c87881f14a84f42bb71b33aacfdaec8cbc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:09.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:09 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:09.570+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:10.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:09.794+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:10.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:09.822+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:10.295 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:10 smithi063 podman[108664]: 2024-04-25 19:08:10.083262216 +0000 UTC m=+6.249319967 container cleanup 490cf821db8fefb2c97431142ff1c87881f14a84f42bb71b33aacfdaec8cbc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-25T19:08:10.295 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:10 smithi063 bash[108664]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3 2024-04-25T19:08:10.296 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:09 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:09.953+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:10.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:10 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:10.022+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:10.546 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:10 smithi063 podman[108696]: 2024-04-25 19:08:10.292737295 +0000 UTC m=+0.630879732 container remove 490cf821db8fefb2c97431142ff1c87881f14a84f42bb71b33aacfdaec8cbc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T19:08:10.809 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:10.539+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:10.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:10 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:10.543+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:11.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:10.807+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:11.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:10.842+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:11.315 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.057444233 +0000 UTC m=+0.036817616 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:11.315 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.168832385 +0000 UTC m=+0.148205805 container create 7bc6e438ea40527c673aa50d7f417d901ec1dab2697dee888b833920d14d3b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:11.316 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:10 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:10.967+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:11.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:10 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:10.987+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:11.577 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.439014498 +0000 UTC m=+0.418387925 container init 7bc6e438ea40527c673aa50d7f417d901ec1dab2697dee888b833920d14d3b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:11.577 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.450856168 +0000 UTC m=+0.430229608 container start 7bc6e438ea40527c673aa50d7f417d901ec1dab2697dee888b833920d14d3b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-25T19:08:11.577 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.520947071 +0000 UTC m=+0.500320498 container attach 7bc6e438ea40527c673aa50d7f417d901ec1dab2697dee888b833920d14d3b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:11.812 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:11.537+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:11.812 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:11.765+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:11.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:11 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:11.573+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:12.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:11.810+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:12.304 INFO:journalctl@ceph.osd.3.smithi063.stdout:Apr 25 19:08:11 smithi063 podman[108784]: 2024-04-25 19:08:11.927332837 +0000 UTC m=+0.906706254 container died 7bc6e438ea40527c673aa50d7f417d901ec1dab2697dee888b833920d14d3b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T19:08:12.304 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:12 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:12.002+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:12.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:12 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:12.000+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:12.683 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T19:08:12.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:12.806 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T19:08:12.806 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T19:08:12.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.4 2024-04-25T19:08:12.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:12.549+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:12.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:12.791+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:12.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:12 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:12.563+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:13.182 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:12 smithi070 systemd[1]: Stopping Ceph osd.4 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:13.183 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:12 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:12.984+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:13.317 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:12.851+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:13.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:13 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:13.020+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:13.454 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:13 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:13.180+0000 7f4d0d2a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:13.454 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:13 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:13.180+0000 7f4d0d2a9700 -1 osd.4 132 *** Got signal Terminated *** 2024-04-25T19:08:13.454 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:13 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:13.180+0000 7f4d0d2a9700 -1 osd.4 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:13.808 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:13.507+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:13.808 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:13.753+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:13.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:13 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:13.602+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:14.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:13.806+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:14.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:13.806+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:14.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:14.042+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:14.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:14.042+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:14.454 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:14.260+0000 7f4d0989d700 -1 osd.4 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:50.156941+0000 front 2024-04-25T19:07:50.156971+0000 (oldest deadline 2024-04-25T19:08:13.557707+0000) 2024-04-25T19:08:14.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:14.018+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:14.736 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:14.466+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:14.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:14 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:14.592+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:15.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:14.734+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:15.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:14.849+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:15.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:14.849+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:15.293 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:14 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:14.980+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:15.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:15.058+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:15.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:15.058+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:15.642 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:15 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:15.291+0000 7f4d0989d700 -1 osd.4 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:50.156941+0000 front 2024-04-25T19:07:50.156971+0000 (oldest deadline 2024-04-25T19:08:13.557707+0000) 2024-04-25T19:08:15.756 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:15.433+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:15.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:15 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:15.620+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:15.954 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:15 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:15.640+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:16.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:15.753+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:16.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:15.753+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:16.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:15.820+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:16.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:15.820+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:16.266 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:15 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:15.959+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:16.267 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:15 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:15.959+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:16.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:16.037+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:16.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:16.037+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:16.672 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:16 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:16.265+0000 7f4d0989d700 -1 osd.4 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:50.156941+0000 front 2024-04-25T19:07:50.156971+0000 (oldest deadline 2024-04-25T19:08:13.557707+0000) 2024-04-25T19:08:16.756 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:16.461+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:16.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:16.599+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:16.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:16 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:16.599+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:16.954 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:16 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:16.670+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:17.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:16.824+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:17.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:16.824+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:17.068 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:16.753+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:17.068 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:16.753+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:17.306 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:16 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:16.983+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:17.307 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:16 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:16.983+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:17.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:17 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:17.005+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:17.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:17 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:17.005+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:17.704 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:17 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4[44105]: 2024-04-25T19:08:17.305+0000 7f4d0989d700 -1 osd.4 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:50.156941+0000 front 2024-04-25T19:07:50.156971+0000 (oldest deadline 2024-04-25T19:08:13.557707+0000) 2024-04-25T19:08:17.776 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:17.434+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:17.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:17 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:17.556+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:17.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:17 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:17.556+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:17.985 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:17 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:17.708+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:18.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:17.775+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:18.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:17.775+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:18.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:17.816+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:18.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:17 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:17.816+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:18.377 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:18.000+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:18.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:18.000+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:18.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:17 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:17.984+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:18.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:17 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:17.984+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:18.761 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:18 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:18.481+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:18.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:18.518+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:18.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:18.518+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:18.954 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:18 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:18.681+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:18.954 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:18 smithi070 podman[73226]: 2024-04-25 19:08:18.755517241 +0000 UTC m=+5.748362412 container died 954fafa52a0f81d29aa359610082ea5fae1d35b205f49c481b7b7be6958cdb6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T19:08:19.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:18 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:18.800+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:19.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:18 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:18.800+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:19.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:18 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:18.759+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:19.067 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:18 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:18.759+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:19.377 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:18.954+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:19.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:18 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:18.954+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:19.449 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73226]: 2024-04-25 19:08:19.113547708 +0000 UTC m=+6.106392886 container cleanup 954fafa52a0f81d29aa359610082ea5fae1d35b205f49c481b7b7be6958cdb6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:19.449 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 bash[73226]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4 2024-04-25T19:08:19.450 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:18 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:18.971+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:19.450 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:18 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:18.971+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:19.704 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73246]: 2024-04-25 19:08:19.448069628 +0000 UTC m=+0.687292995 container remove 954fafa52a0f81d29aa359610082ea5fae1d35b205f49c481b7b7be6958cdb6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:19.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:19.661+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:19.772 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:19.505+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:19.773 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:19.719+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:19.773 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:19.719+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:19.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:19.517+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:19.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:19.517+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:20.052 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:19.996+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:20.052 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:19 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:19.996+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:20.052 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73333]: 2024-04-25 19:08:19.675700875 +0000 UTC m=+0.017877308 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:20.052 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73333]: 2024-04-25 19:08:19.773911534 +0000 UTC m=+0.116087962 container create 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True) 2024-04-25T19:08:20.053 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73333]: 2024-04-25 19:08:19.967703289 +0000 UTC m=+0.309879716 container init 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-25T19:08:20.053 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:19 smithi070 podman[73333]: 2024-04-25 19:08:19.970645151 +0000 UTC m=+0.312821579 container start 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:20.053 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:20 smithi070 podman[73333]: 2024-04-25 19:08:20.050948594 +0000 UTC m=+0.393125041 container attach 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T19:08:20.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:19.770+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:20.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:19 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:19.770+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:20.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:19.935+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:20.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:19.935+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:20.454 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:20 smithi070 podman[73333]: 2024-04-25 19:08:20.254393558 +0000 UTC m=+0.596569988 container died 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T19:08:20.806 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:20 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:20.694+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:20.806 INFO:journalctl@ceph.osd.4.smithi070.stdout:Apr 25 19:08:20 smithi070 podman[73359]: 2024-04-25 19:08:20.787189308 +0000 UTC m=+0.527808380 container remove 97c1142ade89deef0451ec7cca7ac6858a440856fcf01d8ce2afab845d15d2a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T19:08:20.811 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T19:08:20.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:20.540+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:20.817 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:20.744+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:20.818 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:20.744+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:20.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:20.744+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:20.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:20 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:20.744+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:20.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:20.850 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T19:08:20.850 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T19:08:20.850 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.5 2024-04-25T19:08:20.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:20 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:20.559+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:20.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:20 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:20.559+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:21.143 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:20 smithi114 systemd[1]: Stopping Ceph osd.5 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:21.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:21 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:21.044+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:21.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:21 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:21.044+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:21.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:20 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:20.907+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:21.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:20 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:20.907+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:21.502 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:21.141+0000 7f0ea2828700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:21.502 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:21.141+0000 7f0ea2828700 -1 osd.5 132 *** Got signal Terminated *** 2024-04-25T19:08:21.502 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:21.141+0000 7f0ea2828700 -1 osd.5 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:21.761 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:21.499+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:21.761 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:21.710+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:21.761 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:21.710+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:21.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:21.586+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:21.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:21.586+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:22.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:21.759+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:22.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:21 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:21.759+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:22.095 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:21 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:21.703+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:22.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:21.947+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:22.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:21 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:21.947+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:22.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:22.094+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:22.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:22.094+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:22.786 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:22.506+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:22.786 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:22.506+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:22.786 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:22.683+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:22.786 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:22.683+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:22.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:22 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:22.552+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:22.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:22 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:22.552+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:23.050 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:22 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:22.733+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:23.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:22.784+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:23.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:22 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:22.784+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:23.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:22 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:22.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:23.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:22 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:22.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:23.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:22 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:22.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:23.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:23.049+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:23.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:23.049+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:23.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:23.640+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:23.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:23.640+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:23.818 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:23.780+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:23.818 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:23.780+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:23.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:23.545+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:23.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:23 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:23.545+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:23.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:23.575+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:23.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:23.575+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:24.065 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:23 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:23.745+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:24.322 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:24.086+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:24.322 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:24.086+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:24.323 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:24.086+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:24.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:23.954+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:24.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:23.954+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:24.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:23 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:23.954+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:24.608 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:24.547+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:24.608 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:24.547+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:24.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:24.546+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:24.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:24.546+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:24.919 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:24.606+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:24.920 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:24.606+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:24.920 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:24.773+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:24.920 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:24.773+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:24.920 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:24 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:24.773+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.802018+0000 front 2024-04-25T19:08:00.801999+0000 (oldest deadline 2024-04-25T19:08:24.302143+0000) 2024-04-25T19:08:25.066 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:24 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:24.770+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:25.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:24.951+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:25.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:24.951+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:25.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:24.951+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:25.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:25.063+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:25.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:25.063+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:25.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:25.063+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:25.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:25.535+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:25.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:25.535+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:25.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:25.535+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:26.067 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:25.594+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:26.068 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:25.594+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:26.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:25.739+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:39.997634+0000 front 2024-04-25T19:07:39.997427+0000 (oldest deadline 2024-04-25T19:08:05.897110+0000) 2024-04-25T19:08:26.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:25.739+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:48.698930+0000 front 2024-04-25T19:07:48.699173+0000 (oldest deadline 2024-04-25T19:08:13.398966+0000) 2024-04-25T19:08:26.068 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5[44784]: 2024-04-25T19:08:25.739+0000 7f0e9ae14700 -1 osd.5 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.802018+0000 front 2024-04-25T19:08:00.801999+0000 (oldest deadline 2024-04-25T19:08:24.302143+0000) 2024-04-25T19:08:26.068 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:25.632+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:26.069 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:25 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:25.632+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:26.099 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:25 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:25.734+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:26.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:25.997+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:26.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:25.997+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:26.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:25 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:25.997+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:26.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:26.097+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:26.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:26.097+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:26.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:26.097+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:26.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:26.546+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:26.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:26.546+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:26.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:26 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:26.546+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:26.919 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:26 smithi114 podman[73076]: 2024-04-25 19:08:26.622854142 +0000 UTC m=+5.582449707 container died 3c25df63a4af166194cf4368073ece803babacc72ec1ef9a6bca8feb842014e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:26.920 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:26.642+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:26.920 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:26.642+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:26.920 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:26.620+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:26.920 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:26.620+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:26.921 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:26 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:26.620+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:27.142 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:26.781+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:27.142 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:26 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:26.781+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:27.244 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73076]: 2024-04-25 19:08:27.017265469 +0000 UTC m=+5.976861043 container cleanup 3c25df63a4af166194cf4368073ece803babacc72ec1ef9a6bca8feb842014e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:27.244 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 bash[73076]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5 2024-04-25T19:08:27.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:27.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:27.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:27.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:27.140+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:27.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:27.140+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:27.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:27.140+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:27.558 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73095]: 2024-04-25 19:08:27.24333112 +0000 UTC m=+0.622067897 container remove 3c25df63a4af166194cf4368073ece803babacc72ec1ef9a6bca8feb842014e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-25T19:08:27.809 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73187]: 2024-04-25 19:08:27.46062105 +0000 UTC m=+0.018394399 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:27.809 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73187]: 2024-04-25 19:08:27.560844441 +0000 UTC m=+0.118617823 container create c36ad95cc132619f346cb00fe437ce52eef7ef72609ded73c30f2ab598448cf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:27.809 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73187]: 2024-04-25 19:08:27.805322259 +0000 UTC m=+0.363095600 container init c36ad95cc132619f346cb00fe437ce52eef7ef72609ded73c30f2ab598448cf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:08:27.810 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73187]: 2024-04-25 19:08:27.80840334 +0000 UTC m=+0.366176680 container start c36ad95cc132619f346cb00fe437ce52eef7ef72609ded73c30f2ab598448cf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T19:08:27.810 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:27.602+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:27.810 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:27.602+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:27.810 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:27.630+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:27.810 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:27.630+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:27.811 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:27 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:27.630+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:27.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:27.519+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:27.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:27.519+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:27.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:27.519+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:28.067 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:27 smithi114 podman[73187]: 2024-04-25 19:08:27.905838852 +0000 UTC m=+0.463612201 container attach c36ad95cc132619f346cb00fe437ce52eef7ef72609ded73c30f2ab598448cf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:08:28.162 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:27.820+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:28.163 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:27 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:27.820+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:28.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.994+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:28.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.994+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:28.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:27 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:27.994+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:28.434 INFO:journalctl@ceph.osd.5.smithi114.stdout:Apr 25 19:08:28 smithi114 podman[73187]: 2024-04-25 19:08:28.095313858 +0000 UTC m=+0.653087207 container died c36ad95cc132619f346cb00fe437ce52eef7ef72609ded73c30f2ab598448cf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T19:08:28.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:28.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:28.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:28.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:28.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:28.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:28.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:28.581+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:28.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:28.581+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:28.719 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:28.584+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:28.719 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:28.584+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:28.719 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:28 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:28.584+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:28.743 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T19:08:28.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:28.782 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T19:08:28.782 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T19:08:28.782 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.6 2024-04-25T19:08:28.865 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:28.493+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:28.865 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:28.493+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:28.865 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:28.493+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:29.128 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:28 smithi063 systemd[1]: Stopping Ceph osd.6 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:29.128 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:28.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:29.128 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:28.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:29.129 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:28.996+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:29.198 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:28.816+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:29.198 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:28 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:28.816+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:29.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:29.196+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:29.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:29.196+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:29.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:29.196+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:29.628 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.359+0000 7f6483200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:29.628 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.359+0000 7f6483200700 -1 osd.6 132 *** Got signal Terminated *** 2024-04-25T19:08:29.628 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.359+0000 7f6483200700 -1 osd.6 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:29.629 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.444+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:29.629 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.444+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:29.629 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:29 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:29.444+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:29.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:29.609+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:29.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:29.609+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:29.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:29.545+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:29.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:29.545+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:29.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:29 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:29.545+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:30.152 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:29.856+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:30.152 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:29 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:29.856+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:30.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:30.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:30.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:30.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:30.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:30.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:30.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:30.150+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:30.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:30.150+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:30.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:30.150+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:30.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:30.584+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:30.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:30.584+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:30.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:30.499+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:30.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:30.499+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:30.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:30 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:30.499+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:30.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:30.473+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:30.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:30.473+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:30.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:30 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:30.473+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:31.197 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:30.808+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:31.198 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:30.808+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:31.198 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:30 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:30.808+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:31.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:31.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:31.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:31.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:31.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:31.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:31.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:31.028+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:31.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:31.195+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:31.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:31.195+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:31.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:31.195+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:31.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:31.564+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:31.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:31.564+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:31.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:31.564+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:31.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:31.506+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:31.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:31.506+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:31.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:31 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:31.506+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:31.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:31.490+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:31.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:31.490+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:31.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:31.490+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:32.159 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:31.760+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:32.160 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:31.760+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:32.160 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:31 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:31.760+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:32.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:32.040+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:32.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:32.040+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:32.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:32.040+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:32.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:32.040+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:32.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:32.157+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:32.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:32.157+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:32.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:32.157+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:32.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:32.522+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:32.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:32.522+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:32.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:32.522+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:32.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:32.530+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:32.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:32.530+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:32.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:32 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:32.530+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:32.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:32.527+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:32.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:32.527+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:32.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:32 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:32.527+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:33.169 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:32.788+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:33.169 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:32.788+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:33.169 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:32 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:32.788+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:33.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:33.027+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:33.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:33.027+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:33.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:33.027+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:33.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:33.027+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:33.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:33.167+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:33.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:33.167+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:33.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:33.167+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:33.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:33.482+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:33.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:33.483+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:33.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:33.483+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:33.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:33.559+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:33.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:33.559+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:33.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:33 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:33.559+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:33.877 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:33.537+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.478541+0000 front 2024-04-25T19:07:43.478883+0000 (oldest deadline 2024-04-25T19:08:05.178182+0000) 2024-04-25T19:08:33.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:33.537+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.385580+0000 front 2024-04-25T19:07:51.385687+0000 (oldest deadline 2024-04-25T19:08:16.085834+0000) 2024-04-25T19:08:33.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:33.537+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.490281+0000 front 2024-04-25T19:07:59.490633+0000 (oldest deadline 2024-04-25T19:08:25.390546+0000) 2024-04-25T19:08:33.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:33 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6[56405]: 2024-04-25T19:08:33.537+0000 7f647ea00700 -1 osd.6 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.293462+0000 front 2024-04-25T19:08:08.293535+0000 (oldest deadline 2024-04-25T19:08:32.993814+0000) 2024-04-25T19:08:34.165 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:33.790+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:34.166 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:33.790+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:34.166 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:33 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:33.790+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:34.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:34 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:34.049+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:34.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:34 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:34.049+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:34.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:34 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:34.049+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:34.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:34 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:34.049+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:34.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:34.164+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:34.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:34.164+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:34.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:34.164+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:34.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:34.164+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:34.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:34.532+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:34.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:34.532+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:34.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:34.532+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:34.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:34.531+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:34.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:34.531+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:34.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:34 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:34.531+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:35.163 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:34.824+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:35.163 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:34.824+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:35.163 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:34.824+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:35.268 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:34 smithi063 podman[108888]: 2024-04-25 19:08:34.951551049 +0000 UTC m=+5.717532495 container died bf42b622a0fd53922004e123e05af2cc52ecdd942ca884d7c15ea19beb2d67b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-25T19:08:35.268 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:35 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:35.075+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:35.268 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:35 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:35.075+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:35.269 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:35 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:35.075+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:35.269 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:35 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:35.075+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:35.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:35.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:35.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:35.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:35.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:35.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:35.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:35.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:35.614 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:35 smithi063 podman[108888]: 2024-04-25 19:08:35.384844628 +0000 UTC m=+6.150826068 container cleanup bf42b622a0fd53922004e123e05af2cc52ecdd942ca884d7c15ea19beb2d67b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:35.615 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:35 smithi063 bash[108888]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6 2024-04-25T19:08:35.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:35.558+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:35.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:35.558+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:35.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:35.558+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:35.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:35.491+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:35.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:35.491+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:35.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:35 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:35.491+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:35.878 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:35 smithi063 podman[108907]: 2024-04-25 19:08:35.610969481 +0000 UTC m=+0.644237815 container remove bf42b622a0fd53922004e123e05af2cc52ecdd942ca884d7c15ea19beb2d67b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T19:08:36.176 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:35.827+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:36.176 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:35.827+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:36.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:35 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:35.827+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:36.377 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:36 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:36.101+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:36.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:36 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:36.101+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:36.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:36 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:36.101+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:36.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:36 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:36.101+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:36.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:36.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:36.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:36.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:36.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:36.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:36.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:36.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:36.804 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:36 smithi063 podman[108993]: 2024-04-25 19:08:36.451427317 +0000 UTC m=+0.040357845 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:36.804 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:36 smithi063 podman[108993]: 2024-04-25 19:08:36.55337941 +0000 UTC m=+0.142309949 container create c1436614a624b073bfa3be6f19c7abe743705cf4fa44dd866b9b5e6c90db806d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T19:08:36.804 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:36 smithi063 podman[108993]: 2024-04-25 19:08:36.790140399 +0000 UTC m=+0.379070954 container init c1436614a624b073bfa3be6f19c7abe743705cf4fa44dd866b9b5e6c90db806d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:36.804 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:36 smithi063 podman[108993]: 2024-04-25 19:08:36.801838974 +0000 UTC m=+0.390769526 container start c1436614a624b073bfa3be6f19c7abe743705cf4fa44dd866b9b5e6c90db806d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T19:08:36.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:36.604+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:36.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:36.604+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:36.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:36.604+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:36.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:36.514+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:36.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:36.514+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:36.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:36 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:36.514+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:37.128 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:36 smithi063 podman[108993]: 2024-04-25 19:08:36.888992147 +0000 UTC m=+0.477922731 container attach c1436614a624b073bfa3be6f19c7abe743705cf4fa44dd866b9b5e6c90db806d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-25T19:08:37.196 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:36.855+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:37.196 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:36.855+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:37.197 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:36 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:36.855+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:37.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:37.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:37.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:37.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:37.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:37.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:37.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:37.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:37.624 INFO:journalctl@ceph.osd.6.smithi063.stdout:Apr 25 19:08:37 smithi063 podman[108993]: 2024-04-25 19:08:37.271007474 +0000 UTC m=+0.859937987 container died c1436614a624b073bfa3be6f19c7abe743705cf4fa44dd866b9b5e6c90db806d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:37.625 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:37 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:37.145+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:37.625 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:37 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:37.145+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:37.625 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:37 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:37.145+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:37.625 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:37 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:37.145+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:37.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:37.637+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:37.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:37.637+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:37.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:37.637+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:37.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:37.473+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:37.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:37.473+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:37.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:37 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:37.473+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:38.028 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T19:08:38.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:38.149 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T19:08:38.150 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T19:08:38.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.7 2024-04-25T19:08:38.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:37.862+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:38.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:37.862+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:38.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:37 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:37.862+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:38.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:38.153+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:38.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:38.153+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:38.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:38.153+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:38.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:38 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:38.153+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:38.447 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 systemd[1]: Stopping Ceph osd.7 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:38.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:38.241+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:38.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:38.241+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:38.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:38.241+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:38.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:38.241+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:38.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.446+0000 7f55fa565700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:38.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.446+0000 7f55fa565700 -1 osd.7 132 *** Got signal Terminated *** 2024-04-25T19:08:38.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.446+0000 7f55fa565700 -1 osd.7 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:38.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:38.595+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:38.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:38.595+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:38.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:38.595+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:38.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:38.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:38.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:38.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:38.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:38 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:38.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:39.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.833+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:39.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.833+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:39.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:38 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:38.833+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:39.627 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:39 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:39.173+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:39.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:39 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:39.173+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:39.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:39 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:39.173+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:39.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:39 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:39.173+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:39.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:39.234+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:39.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:39.234+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:39.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:39.234+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:39.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:39.234+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:39.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:39.629+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:39.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:39.629+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:39.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:39.629+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:39.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:39.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:39.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:39.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:39.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:39 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:39.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:40.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:39.841+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:40.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:39.841+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:40.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:39.841+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:40.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:40 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:40.149+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:40.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:40 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:40.149+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:40.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:40 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:40.149+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:40.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:40 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:40.149+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:40.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:40.212+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:40.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:40.212+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:40.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:40.212+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:40.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:40.212+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:40.705 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:40.212+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:40.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:40.591+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:40.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:40.591+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:40.591+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:40.591+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:40.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:40.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:40.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:40.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:40 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:40.504+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:41.171 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:40.873+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:41.172 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:40.873+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:41.172 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:40 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:40.873+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:41.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:41.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:41.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:41.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:41.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:41.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:41.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:41.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:41.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:41.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:41.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:41 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:41.187+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:41.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:41 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:41.187+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:41.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:41 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:41.187+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:41.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:41 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:41.187+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:41.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:41.560+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:41.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:41.560+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:41.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:41.560+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:41.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:41.560+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:41.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:41.502+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:41.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:41.502+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:41.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:41.502+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:41.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:41 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:41.502+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:42.204 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:41.865+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:42.219 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:41.865+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:42.220 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:41.865+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:42.220 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:41 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:41.865+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.542868+0000 front 2024-04-25T19:08:17.542850+0000 (oldest deadline 2024-04-25T19:08:41.643072+0000) 2024-04-25T19:08:42.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:42.179+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:42.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:42.179+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:42.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:42.179+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:42.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:42.179+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:42.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:42 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:42.179+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:42.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:42.209+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:42.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:42.209+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:42.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:42.209+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:42.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:42.209+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:42.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:42.209+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:42.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:42.514+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:42.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:42.514+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:42.514+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:42.514+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:42.509+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:42.509+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:42.509+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:42.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:42 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:42.509+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:43.176 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:42.892+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.739376+0000 front 2024-04-25T19:07:51.739406+0000 (oldest deadline 2024-04-25T19:08:15.239575+0000) 2024-04-25T19:08:43.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:42.892+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:59.940981+0000 front 2024-04-25T19:07:59.940860+0000 (oldest deadline 2024-04-25T19:08:25.840865+0000) 2024-04-25T19:08:43.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:42.892+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:05.841881+0000 front 2024-04-25T19:08:05.841776+0000 (oldest deadline 2024-04-25T19:08:30.541449+0000) 2024-04-25T19:08:43.177 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:42 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7[51096]: 2024-04-25T19:08:42.892+0000 7f55f2b51700 -1 osd.7 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.542868+0000 front 2024-04-25T19:08:17.542850+0000 (oldest deadline 2024-04-25T19:08:41.643072+0000) 2024-04-25T19:08:43.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:43.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:43.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:43.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:43.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:43.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:43.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:43.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:43.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:43.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:43.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:43.146+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:43.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:43.146+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:43.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:43.146+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:43.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:43.146+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:43.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:43 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:43.146+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:43.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:43.473+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:43.817 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:43.473+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:43.473+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:43.473+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:43.480+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:43.480+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:43.480+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:43.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:43 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:43.480+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:44.263 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:43 smithi070 podman[73449]: 2024-04-25 19:08:43.974793815 +0000 UTC m=+5.636783129 container died 599dd8e7cee79102d8559e1480cf276abd96e076261e7714d04ed6ade3fc2464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:08:44.264 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:44 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:44.129+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:44.264 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:44 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:44.129+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:44.264 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:44 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:44.129+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:44.264 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:44 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:44.129+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:44.264 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:44 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:44.129+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:44.613 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:44 smithi070 podman[73449]: 2024-04-25 19:08:44.370129811 +0000 UTC m=+6.032119104 container cleanup 599dd8e7cee79102d8559e1480cf276abd96e076261e7714d04ed6ade3fc2464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0) 2024-04-25T19:08:44.614 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:44 smithi070 bash[73449]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7 2024-04-25T19:08:44.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:44.158+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:44.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:44.158+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:44.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:44.158+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:44.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:44.158+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:44.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:44 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:44.158+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:44.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:44.440+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:44.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:44.440+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:44.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:44.440+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:44.719 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:44.440+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:44.720 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:44.465+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:44.720 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:44.465+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:44.720 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:44.465+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:44.720 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:44 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:44.465+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:44.922 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:44 smithi070 podman[73466]: 2024-04-25 19:08:44.612886642 +0000 UTC m=+0.640147823 container remove 599dd8e7cee79102d8559e1480cf276abd96e076261e7714d04ed6ade3fc2464 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T19:08:45.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:45.127+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:45.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:45.127+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:45.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:45.127+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:45.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:45.127+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:45.205 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:45 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:45.127+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:45.205 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:44 smithi070 podman[73556]: 2024-04-25 19:08:44.921917979 +0000 UTC m=+0.106893152 container create 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T19:08:45.205 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:44 smithi070 podman[73556]: 2024-04-25 19:08:44.832397393 +0000 UTC m=+0.017372574 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:45.205 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 podman[73556]: 2024-04-25 19:08:45.132797681 +0000 UTC m=+0.317772855 container init 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T19:08:45.205 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 podman[73556]: 2024-04-25 19:08:45.135890159 +0000 UTC m=+0.320865337 container start 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T19:08:45.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:45 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:45.131+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:45.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:45 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:45.131+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:45.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:45 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:45.131+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:45.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:45 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:45.131+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:45.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:45 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:45.131+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:45.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 podman[73556]: 2024-04-25 19:08:45.219546951 +0000 UTC m=+0.404522143 container attach 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T19:08:45.704 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 podman[73556]: 2024-04-25 19:08:45.425731463 +0000 UTC m=+0.610706637 container died 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T19:08:45.734 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:45.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:45.734 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:45.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:45.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:45.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:45.417+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:45.417+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:45.417+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:45.735 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:45 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:45.417+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:45.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T19:08:46.025 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 podman[73582]: 2024-04-25 19:08:45.952005313 +0000 UTC m=+0.517014139 container remove 62939e4cbbbde3bbc883452ce379eeca8c9bc3391f13de717e0b6564d9adeae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-25T19:08:46.026 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-25T19:08:46.026 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 systemd[1]: Stopped Ceph osd.7 for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T19:08:46.026 INFO:journalctl@ceph.osd.7.smithi070.stdout:Apr 25 19:08:45 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.7.service: Consumed 2min 33.817s CPU time. 2024-04-25T19:08:46.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:46.048 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T19:08:46.048 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T19:08:46.048 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.8 2024-04-25T19:08:46.317 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 systemd[1]: Stopping Ceph osd.8 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:46.435 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:46.098+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:46.435 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:46.098+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:46.435 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:46.098+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:46.435 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:46.098+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:46.435 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:46 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:46.098+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:46.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:46.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:46.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:46.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:46.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:46.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:46.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:46.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:46.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:46 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:46.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:46.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:46.382+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:46.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:46.382+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:46.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:46.382+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:46.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:46.382+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:46.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.353+0000 7f17fb40e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:46.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.353+0000 7f17fb40e700 -1 osd.8 132 *** Got signal Terminated *** 2024-04-25T19:08:46.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.353+0000 7f17fb40e700 -1 osd.8 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:46.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.412+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:46.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.412+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:46.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.412+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:46.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:46 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:46.412+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:47.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:47.107+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:47.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:47.107+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:47.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:47.107+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:47.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:47.107+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:47.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:47.107+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:47.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:47.161+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:47.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:47.161+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:47.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:47.161+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:47.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:47.161+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:47.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:47 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:47.161+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:47.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:47.377+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:47.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:47.377+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:47.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:47.377+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:47.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:47.377+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:47.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:47.377+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:21.631901+0000 front 2024-04-25T19:08:21.631928+0000 (oldest deadline 2024-04-25T19:08:46.932035+0000) 2024-04-25T19:08:47.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:47.408+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:47.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:47.408+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:47.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:47.408+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:47.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:47.408+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:47.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:47 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:47.408+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:48.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:48 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:48.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:48.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:48 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:48.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:48.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:48 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:48.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:48.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:48 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:48.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:48.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:48 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:48.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:48.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:48.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:48.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:48.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:48.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:48.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:48 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:48.165+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:48.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:48.416+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:48.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:48.416+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:48.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:48.416+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:48.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:48.416+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:48.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:48.416+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:48.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:48.401+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:48.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:48.401+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:48.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:48.401+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:48.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:48.401+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:48.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:48 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:48.401+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:21.631901+0000 front 2024-04-25T19:08:21.631928+0000 (oldest deadline 2024-04-25T19:08:46.932035+0000) 2024-04-25T19:08:49.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:49 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:49.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:49.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:49 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:49.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:49.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:49 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:49.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:49.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:49 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:49.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:49.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:49 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:49.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:49.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:49.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:49.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:49.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:49.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:49.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:49 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:49.184+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:49.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:49.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:49.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:49.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:49.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:49.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:49.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:49.406+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:49.406+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:49.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:49.406+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:49.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:49.406+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:49.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:49 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:49.406+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:21.631901+0000 front 2024-04-25T19:08:21.631928+0000 (oldest deadline 2024-04-25T19:08:46.932035+0000) 2024-04-25T19:08:50.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:50 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:50.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:50.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:50 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:50.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:50.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:50 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:50.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:50.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:50 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:50.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:50.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:50 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:50.121+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:50.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:50.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:50.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:50.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:50.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:50.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:50 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:50.226+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:50.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:50.366+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:50.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:50.366+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:50.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:50.366+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:50.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:50.366+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:50.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:50.366+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:50.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:50.445+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.723583+0000 front 2024-04-25T19:07:41.723369+0000 (oldest deadline 2024-04-25T19:08:05.223282+0000) 2024-04-25T19:08:50.818 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:50.445+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:57.927138+0000 front 2024-04-25T19:07:57.927253+0000 (oldest deadline 2024-04-25T19:08:22.027267+0000) 2024-04-25T19:08:50.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:50.445+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.928738+0000 front 2024-04-25T19:08:07.928521+0000 (oldest deadline 2024-04-25T19:08:31.428357+0000) 2024-04-25T19:08:50.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:50.445+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.730404+0000 front 2024-04-25T19:08:17.730433+0000 (oldest deadline 2024-04-25T19:08:40.030464+0000) 2024-04-25T19:08:50.819 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:50 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8[51346]: 2024-04-25T19:08:50.445+0000 7f17f39fa700 -1 osd.8 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:21.631901+0000 front 2024-04-25T19:08:21.631928+0000 (oldest deadline 2024-04-25T19:08:46.932035+0000) 2024-04-25T19:08:51.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:51.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:51.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:51.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:51.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:51.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:51 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:51.137+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:51.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:51.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:51.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:51.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:51.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:51.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:51 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:51.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:51.805 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:51 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:51.395+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:51.805 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:51 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:51.395+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:51.805 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:51 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:51.396+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:51.805 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:51 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:51.396+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:51.805 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:51 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:51.396+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:52.067 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:51 smithi114 podman[73300]: 2024-04-25 19:08:51.803684654 +0000 UTC m=+5.565704799 container died 2c1a14a52029b5f5259e2eebc987abc80b8530deccddacde3b6b40b187a08dea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:08:52.408 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:52 smithi114 podman[73300]: 2024-04-25 19:08:52.170691414 +0000 UTC m=+5.932711565 container cleanup 2c1a14a52029b5f5259e2eebc987abc80b8530deccddacde3b6b40b187a08dea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:08:52.408 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:52 smithi114 bash[73300]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8 2024-04-25T19:08:52.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:52.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:52.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:52.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:52.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:52.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:52 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:52.134+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:52.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:52.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:52.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:52.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:52.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:52.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:52 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:52.289+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:52.774 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:52 smithi114 podman[73320]: 2024-04-25 19:08:52.425681548 +0000 UTC m=+0.620975939 container remove 2c1a14a52029b5f5259e2eebc987abc80b8530deccddacde3b6b40b187a08dea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T19:08:52.774 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:52.405+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:52.774 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:52.405+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:52.774 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:52.405+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:52.774 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:52.405+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:52.775 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:52.405+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:53.067 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:52 smithi114 podman[73413]: 2024-04-25 19:08:52.772885836 +0000 UTC m=+0.103217353 container create 241af1f95270a10bf8b6600bcabe4421a12961e41275c6367117995ecaba4284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5) 2024-04-25T19:08:53.068 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:52 smithi114 podman[73413]: 2024-04-25 19:08:52.687363728 +0000 UTC m=+0.017695246 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:08:53.068 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:53 smithi114 podman[73413]: 2024-04-25 19:08:53.001021039 +0000 UTC m=+0.331352574 container init 241af1f95270a10bf8b6600bcabe4421a12961e41275c6367117995ecaba4284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-25T19:08:53.068 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:53 smithi114 podman[73413]: 2024-04-25 19:08:53.004720747 +0000 UTC m=+0.335052265 container start 241af1f95270a10bf8b6600bcabe4421a12961e41275c6367117995ecaba4284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T19:08:53.448 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:53 smithi114 podman[73413]: 2024-04-25 19:08:53.101155821 +0000 UTC m=+0.431487356 container attach 241af1f95270a10bf8b6600bcabe4421a12961e41275c6367117995ecaba4284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-25T19:08:53.449 INFO:journalctl@ceph.osd.8.smithi114.stdout:Apr 25 19:08:53 smithi114 podman[73413]: 2024-04-25 19:08:53.29907232 +0000 UTC m=+0.629403853 container died 241af1f95270a10bf8b6600bcabe4421a12961e41275c6367117995ecaba4284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T19:08:53.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:53.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:53.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:53.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:53.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:53.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:53 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:53.123+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:53.630 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:53 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:53.311+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:53.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:53 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:53.446+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:53.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:53 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:53.446+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:53.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:53 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:53.446+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:53.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:53 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:53.446+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:53.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:53 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:53.446+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:53.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T19:08:54.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:08:54.002 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T19:08:54.002 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T19:08:54.003 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.9 2024-04-25T19:08:54.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 systemd[1]: Stopping Ceph osd.9 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:08:54.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:54.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:54.379 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:54.380 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:54.380 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:54.380 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.312+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:54.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:54.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:54.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:54.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:54.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:54.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:54 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:54.169+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:54.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:54.488+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:54.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:54.488+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:54.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:54.488+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:54.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:54.488+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:54.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:54 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:54.488+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:54.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.618+0000 7f9d9c200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:08:54.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.618+0000 7f9d9c200700 -1 osd.9 132 *** Got signal Terminated *** 2024-04-25T19:08:54.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:54 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:54.618+0000 7f9d9c200700 -1 osd.9 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:08:55.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:55.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:55.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:55.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:55.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:55.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:55 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:55.188+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:55.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:55.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:55.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:55.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:55.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:55.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:55 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:55.276+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:55.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:55.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:55.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:55.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:55.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:55.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:55.489+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:08:56.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:56.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:56.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:56.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:56.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:56.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:56.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:56 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:56.179+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:08:56.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:56.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:56.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:56.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:56.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:56.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:56 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:56.241+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:56.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:56.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:56.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:56.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:56.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:56.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:56 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:56.493+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:08:57.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:57.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:57.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:57.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:57.456 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:57.456 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:57.456 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:57 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:57.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:08:57.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:57.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:57.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:57.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.260+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:57.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.261+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:57.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:57 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:57.261+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:57.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:57.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:57.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:57.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:57.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:57.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:57 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:57.520+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:08:58.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:58.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:58.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:58.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:58.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:58.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:58.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:58 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:58.146+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:08:58.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:58.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:58.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:58.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:58.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:58.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:58 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:58.222+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:58.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:58.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:58.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:58.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:58.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:58.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:58 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:58.497+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:08:59.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:08:59.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:08:59.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:08:59.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:08:59.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:08:59.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:08:59.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:08:59 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:08:59.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:08:59.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:43.521113+0000 front 2024-04-25T19:07:43.520615+0000 (oldest deadline 2024-04-25T19:08:04.620842+0000) 2024-04-25T19:08:59.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:51.528546+0000 front 2024-04-25T19:07:51.528640+0000 (oldest deadline 2024-04-25T19:08:13.828832+0000) 2024-04-25T19:08:59.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:07:56.731370+0000 front 2024-04-25T19:07:56.731172+0000 (oldest deadline 2024-04-25T19:08:22.031407+0000) 2024-04-25T19:08:59.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:07.834838+0000 front 2024-04-25T19:08:07.835175+0000 (oldest deadline 2024-04-25T19:08:30.735123+0000) 2024-04-25T19:08:59.628 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.037194+0000 front 2024-04-25T19:08:16.037249+0000 (oldest deadline 2024-04-25T19:08:41.337490+0000) 2024-04-25T19:08:59.629 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:08:59 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9[62963]: 2024-04-25T19:08:59.199+0000 7f9d92a00700 -1 osd.9 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.239278+0000 front 2024-04-25T19:08:24.239304+0000 (oldest deadline 2024-04-25T19:08:47.739437+0000) 2024-04-25T19:08:59.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:08:59.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:08:59.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:08:59.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:08:59.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:08:59.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:08:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:08:59.501+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:00.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:00.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:00.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:00.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:00.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:00.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:00.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:00 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:00.178+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:00.526 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:00 smithi063 podman[109096]: 2024-04-25 19:09:00.197159753 +0000 UTC m=+5.714162048 container died 6fa27f8ed2886bf02f9aba53471eb98d11eaed8f83ff039a5c03f77c2fcb521c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T19:09:00.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:00.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:00.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:00.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:00.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:00.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:00 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:00.470+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:00.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:00 smithi063 podman[109096]: 2024-04-25 19:09:00.718954029 +0000 UTC m=+6.235956342 container cleanup 6fa27f8ed2886bf02f9aba53471eb98d11eaed8f83ff039a5c03f77c2fcb521c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T19:09:00.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:00 smithi063 bash[109096]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9 2024-04-25T19:09:01.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:00 smithi063 podman[109114]: 2024-04-25 19:09:00.978398733 +0000 UTC m=+0.764029625 container remove 6fa27f8ed2886bf02f9aba53471eb98d11eaed8f83ff039a5c03f77c2fcb521c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-25T19:09:01.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:01.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:01.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:01.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:01.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:01.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:01.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:01 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:01.199+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:01.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:01.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:01.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:01.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:01.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:01.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:01 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:01.500+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:02.101 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:01 smithi063 podman[109201]: 2024-04-25 19:09:01.715450747 +0000 UTC m=+0.034242082 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:09:02.102 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:01 smithi063 podman[109201]: 2024-04-25 19:09:01.829136021 +0000 UTC m=+0.147927388 container create bc4ee0b5460516cbdc592518397ff32d2fb4fc9b8b2bacc91b8d5ed99ceecd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD) 2024-04-25T19:09:02.102 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:02 smithi063 podman[109201]: 2024-04-25 19:09:02.099497079 +0000 UTC m=+0.418288453 container init bc4ee0b5460516cbdc592518397ff32d2fb4fc9b8b2bacc91b8d5ed99ceecd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:09:02.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:02 smithi063 podman[109201]: 2024-04-25 19:09:02.111201502 +0000 UTC m=+0.429992863 container start bc4ee0b5460516cbdc592518397ff32d2fb4fc9b8b2bacc91b8d5ed99ceecd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:09:02.378 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:02 smithi063 podman[109201]: 2024-04-25 19:09:02.19798979 +0000 UTC m=+0.516781207 container attach bc4ee0b5460516cbdc592518397ff32d2fb4fc9b8b2bacc91b8d5ed99ceecd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-25T19:09:02.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:02.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:02.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:02.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:02.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:02.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:02.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:02 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:02.194+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:02.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:02.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:02.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:02.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:02.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:02.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:02 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:02.458+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:02.878 INFO:journalctl@ceph.osd.9.smithi063.stdout:Apr 25 19:09:02 smithi063 podman[109227]: 2024-04-25 19:09:02.647203698 +0000 UTC m=+0.040148994 container died bc4ee0b5460516cbdc592518397ff32d2fb4fc9b8b2bacc91b8d5ed99ceecd86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:09:03.373 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T19:09:03.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.176+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.176+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.176+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.176+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:03.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.177+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:03.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:03.469 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T19:09:03.470 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T19:09:03.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.10 2024-04-25T19:09:03.762 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 systemd[1]: Stopping Ceph osd.10 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:03.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:03.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:03.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:03.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:03.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:03.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:03.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:04.184 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.759+0000 7f73a631e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:04.185 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.759+0000 7f73a631e700 -1 osd.10 132 *** Got signal Terminated *** 2024-04-25T19:09:04.185 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:03 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:03.759+0000 7f73a631e700 -1 osd.10 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:09:04.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.181+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:04.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.181+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:04.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.182+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:04.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.182+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:04.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.182+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:04.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.182+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:04.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:04 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:04.182+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:04.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:04.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:04.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:04.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:04.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:04.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:04 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:04.427+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:05.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:05.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:05.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:05.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:05.705 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:05.705 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:05.705 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:05 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:05.222+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:05.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:05.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:05.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:05.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:05.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:05.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:05 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:05.420+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:06.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:06.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:06.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:06 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:06.174+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:06.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:06.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:06.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:06.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:06.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:06.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:06.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:06 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:06.414+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:07.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:07.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:07.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:07.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:07.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:07.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:07.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:07 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:07.126+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:07.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:07.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:07 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:07.436+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:08.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:41.313089+0000 front 2024-04-25T19:07:41.313132+0000 (oldest deadline 2024-04-25T19:08:04.812842+0000) 2024-04-25T19:09:08.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:52.314865+0000 front 2024-04-25T19:07:52.314898+0000 (oldest deadline 2024-04-25T19:08:15.215051+0000) 2024-04-25T19:09:08.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.516283+0000 front 2024-04-25T19:08:00.516201+0000 (oldest deadline 2024-04-25T19:08:23.416308+0000) 2024-04-25T19:09:08.454 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6814 osd.3 since back 2024-04-25T19:08:08.117855+0000 front 2024-04-25T19:08:08.118031+0000 (oldest deadline 2024-04-25T19:08:33.417688+0000) 2024-04-25T19:09:08.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:16.318544+0000 front 2024-04-25T19:08:16.318513+0000 (oldest deadline 2024-04-25T19:08:39.818624+0000) 2024-04-25T19:09:08.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:24.519625+0000 front 2024-04-25T19:08:24.519557+0000 (oldest deadline 2024-04-25T19:08:50.419377+0000) 2024-04-25T19:09:08.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:08 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10[58133]: 2024-04-25T19:09:08.161+0000 7f739e90a700 -1 osd.10 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:30.420044+0000 front 2024-04-25T19:08:30.420163+0000 (oldest deadline 2024-04-25T19:08:55.719811+0000) 2024-04-25T19:09:08.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:08.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:08.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:08.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:08.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:08.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:08.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:08 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:08.438+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:09.533 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:09 smithi070 podman[73674]: 2024-04-25 19:09:09.23850483 +0000 UTC m=+5.581139904 container died 9b2b2271385be5bd70af8ac3acfed5fc9ac135b0401850d3e857b2faad55ab93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:09:09.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:09.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:09 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:09.467+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:09.879 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:09 smithi070 podman[73674]: 2024-04-25 19:09:09.602146743 +0000 UTC m=+5.944781815 container cleanup 9b2b2271385be5bd70af8ac3acfed5fc9ac135b0401850d3e857b2faad55ab93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T19:09:09.879 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:09 smithi070 bash[73674]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10 2024-04-25T19:09:10.204 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:09 smithi070 podman[73691]: 2024-04-25 19:09:09.878099474 +0000 UTC m=+0.629740733 container remove 9b2b2271385be5bd70af8ac3acfed5fc9ac135b0401850d3e857b2faad55ab93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-25T19:09:10.533 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.129546185 +0000 UTC m=+0.017691729 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:09:10.534 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.255182943 +0000 UTC m=+0.143328520 container create 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T19:09:10.534 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.449211143 +0000 UTC m=+0.337356679 container init 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T19:09:10.534 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.452425381 +0000 UTC m=+0.340570917 container start 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-25T19:09:10.534 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.532490282 +0000 UTC m=+0.420635834 container attach 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-25T19:09:10.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:10.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:10 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:10.445+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:10.954 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:10 smithi070 podman[73781]: 2024-04-25 19:09:10.748939102 +0000 UTC m=+0.637084649 container died 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux ) 2024-04-25T19:09:11.294 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:11 smithi070 podman[73807]: 2024-04-25 19:09:11.277199924 +0000 UTC m=+0.523215449 container remove 51fe50854164f22d589f9048781f370153eb717c533f62e43cb5c46965c8b2f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True) 2024-04-25T19:09:11.295 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:11 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.10.service: Deactivated successfully. 2024-04-25T19:09:11.295 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:11 smithi070 systemd[1]: Stopped Ceph osd.10 for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T19:09:11.295 INFO:journalctl@ceph.osd.10.smithi070.stdout:Apr 25 19:09:11 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.10.service: Consumed 2min 15.954s CPU time. 2024-04-25T19:09:11.296 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T19:09:11.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:11.334 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T19:09:11.334 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T19:09:11.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.11 2024-04-25T19:09:11.616 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 systemd[1]: Stopping Ceph osd.11 for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:11.617 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.406+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:11.618 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.614+0000 7feb943df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:11.618 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.614+0000 7feb943df700 -1 osd.11 132 *** Got signal Terminated *** 2024-04-25T19:09:11.618 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:11 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:11.614+0000 7feb943df700 -1 osd.11 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T19:09:12.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:12.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:12.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:12 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:12.377+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6820 osd.8 since back 2024-04-25T19:08:46.301675+0000 front 2024-04-25T19:08:46.301695+0000 (oldest deadline 2024-04-25T19:09:12.201683+0000) 2024-04-25T19:09:13.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:13.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:13 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:13.359+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6820 osd.8 since back 2024-04-25T19:08:46.301675+0000 front 2024-04-25T19:08:46.301695+0000 (oldest deadline 2024-04-25T19:09:12.201683+0000) 2024-04-25T19:09:14.722 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:14.722 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:14.722 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:14.723 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:14.723 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:14.723 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:14.723 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:14.723 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:14 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:14.322+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6820 osd.8 since back 2024-04-25T19:08:46.301675+0000 front 2024-04-25T19:08:46.301695+0000 (oldest deadline 2024-04-25T19:09:12.201683+0000) 2024-04-25T19:09:15.734 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:15.734 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:15.734 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:15.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:15.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:15.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:15.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:15.735 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:15 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:15.331+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6820 osd.8 since back 2024-04-25T19:08:46.301675+0000 front 2024-04-25T19:08:46.301695+0000 (oldest deadline 2024-04-25T19:09:12.201683+0000) 2024-04-25T19:09:16.817 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-04-25T19:07:42.989486+0000 front 2024-04-25T19:07:42.989409+0000 (oldest deadline 2024-04-25T19:08:05.289351+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-04-25T19:07:47.590512+0000 front 2024-04-25T19:07:47.590430+0000 (oldest deadline 2024-04-25T19:08:15.191263+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2024-04-25T19:08:00.392802+0000 front 2024-04-25T19:08:00.392847+0000 (oldest deadline 2024-04-25T19:08:26.292941+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-04-25T19:08:17.995427+0000 front 2024-04-25T19:08:17.995395+0000 (oldest deadline 2024-04-25T19:08:39.695516+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6812 osd.5 since back 2024-04-25T19:08:23.196418+0000 front 2024-04-25T19:08:23.196517+0000 (oldest deadline 2024-04-25T19:08:46.696560+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.63:6822 osd.6 since back 2024-04-25T19:08:33.499636+0000 front 2024-04-25T19:08:33.499556+0000 (oldest deadline 2024-04-25T19:08:55.199288+0000) 2024-04-25T19:09:16.818 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-04-25T19:08:41.001027+0000 front 2024-04-25T19:08:41.001048+0000 (oldest deadline 2024-04-25T19:09:06.300946+0000) 2024-04-25T19:09:16.819 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:16 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11[57949]: 2024-04-25T19:09:16.376+0000 7feb8c9cb700 -1 osd.11 132 heartbeat_check: no reply from 172.21.15.114:6820 osd.8 since back 2024-04-25T19:08:46.301675+0000 front 2024-04-25T19:08:46.301695+0000 (oldest deadline 2024-04-25T19:09:12.201683+0000) 2024-04-25T19:09:17.300 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:17 smithi114 podman[73519]: 2024-04-25 19:09:17.027582471 +0000 UTC m=+5.508504267 container died 88ea9569afac473f5fb9d065484f61017a1eb99bf2753a846d8f2165a010881a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416) 2024-04-25T19:09:17.567 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:17 smithi114 podman[73519]: 2024-04-25 19:09:17.390820558 +0000 UTC m=+5.871742345 container cleanup 88ea9569afac473f5fb9d065484f61017a1eb99bf2753a846d8f2165a010881a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:09:17.568 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:17 smithi114 bash[73519]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11 2024-04-25T19:09:17.998 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:17 smithi114 podman[73548]: 2024-04-25 19:09:17.677979596 +0000 UTC m=+0.645770347 container remove 88ea9569afac473f5fb9d065484f61017a1eb99bf2753a846d8f2165a010881a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-25T19:09:18.304 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:17 smithi114 podman[73636]: 2024-04-25 19:09:17.89975639 +0000 UTC m=+0.017869677 image pull da4b004c09e820713483d039bc76fbd660be678f2ea3c03e059a6161b74dab76 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4 2024-04-25T19:09:18.305 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:18 smithi114 podman[73636]: 2024-04-25 19:09:18.00095235 +0000 UTC m=+0.119065626 container create 8d32ec6a1a84e9c9c26b77ae78cb7bc4e70e2fc6df493474635fb9615d53007d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:09:18.305 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:18 smithi114 podman[73636]: 2024-04-25 19:09:18.221143091 +0000 UTC m=+0.339256363 container init 8d32ec6a1a84e9c9c26b77ae78cb7bc4e70e2fc6df493474635fb9615d53007d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T19:09:18.305 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:18 smithi114 podman[73636]: 2024-04-25 19:09:18.224363844 +0000 UTC m=+0.342477115 container start 8d32ec6a1a84e9c9c26b77ae78cb7bc4e70e2fc6df493474635fb9615d53007d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:09:18.305 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:18 smithi114 podman[73636]: 2024-04-25 19:09:18.303467308 +0000 UTC m=+0.421580590 container attach 8d32ec6a1a84e9c9c26b77ae78cb7bc4e70e2fc6df493474635fb9615d53007d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T19:09:18.567 INFO:journalctl@ceph.osd.11.smithi114.stdout:Apr 25 19:09:18 smithi114 podman[73636]: 2024-04-25 19:09:18.520202872 +0000 UTC m=+0.638316150 container died 8d32ec6a1a84e9c9c26b77ae78cb7bc4e70e2fc6df493474635fb9615d53007d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-25T19:09:19.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T19:09:19.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:19.129 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T19:09:19.130 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T19:09:19.130 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.a 2024-04-25T19:09:19.627 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:19 smithi063 systemd[1]: Stopping Ceph mds.a for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:20.128 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a[70332]: 2024-04-25T19:09:19.769+0000 7fa23ac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:20.128 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:19 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a[70332]: 2024-04-25T19:09:19.769+0000 7fa23ac00700 -1 mds.a *** got signal Terminated *** 2024-04-25T19:09:22.487 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:22 smithi063 podman[109317]: 2024-04-25 19:09:22.201354745 +0000 UTC m=+2.586298903 container died 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-25T19:09:22.878 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:22 smithi063 podman[109317]: 2024-04-25 19:09:22.669288923 +0000 UTC m=+3.054233081 container cleanup 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T19:09:22.878 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:22 smithi063 bash[109317]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a 2024-04-25T19:09:23.377 INFO:journalctl@ceph.mds.a.smithi063.stdout:Apr 25 19:09:22 smithi063 podman[109331]: 2024-04-25 19:09:22.878770657 +0000 UTC m=+0.658292027 container remove 1323d794ac39f30c91a414ce133be8038a4d960bea77939e160518ed544f11e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:09:23.446 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T19:09:23.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:23.561 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T19:09:23.561 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T19:09:23.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.d 2024-04-25T19:09:24.128 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:23 smithi063 systemd[1]: Stopping Ceph mds.d for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:24.627 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d[69155]: 2024-04-25T19:09:24.188+0000 7f6adca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:24.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:24 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d[69155]: 2024-04-25T19:09:24.188+0000 7f6adca00700 -1 mds.d *** got signal Terminated *** 2024-04-25T19:09:26.862 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:26 smithi063 podman[109436]: 2024-04-25 19:09:26.578254697 +0000 UTC m=+2.523439551 container died 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:09:27.128 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:26 smithi063 podman[109436]: 2024-04-25 19:09:26.981261292 +0000 UTC m=+2.926446139 container cleanup 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T19:09:27.128 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:26 smithi063 bash[109436]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d 2024-04-25T19:09:27.628 INFO:journalctl@ceph.mds.d.smithi063.stdout:Apr 25 19:09:27 smithi063 podman[109450]: 2024-04-25 19:09:27.157072627 +0000 UTC m=+0.565743139 container remove 0dd4e385d17bfe256827b23b35257155494dfe2c717499e4368ea6d576263ce0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T19:09:27.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T19:09:27.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:27.813 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T19:09:27.813 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T19:09:27.813 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.g 2024-04-25T19:09:28.378 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:27 smithi063 systemd[1]: Stopping Ceph mds.g for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:28.878 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g[69924]: 2024-04-25T19:09:28.466+0000 7f9951600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:28.878 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:28 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g[69924]: 2024-04-25T19:09:28.466+0000 7f9951600700 -1 mds.g *** got signal Terminated *** 2024-04-25T19:09:29.841 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:29 smithi063 podman[109557]: 2024-04-25 19:09:29.540726522 +0000 UTC m=+1.210968690 container died 750410dee27fe8074654c22e23d8564f96c70f73cb8e9b2f946bde7709a0fd1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T19:09:30.128 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:29 smithi063 podman[109557]: 2024-04-25 19:09:29.96001884 +0000 UTC m=+1.630261029 container cleanup 750410dee27fe8074654c22e23d8564f96c70f73cb8e9b2f946bde7709a0fd1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T19:09:30.128 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:29 smithi063 bash[109557]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g 2024-04-25T19:09:30.620 INFO:journalctl@ceph.mds.g.smithi063.stdout:Apr 25 19:09:30 smithi063 podman[109571]: 2024-04-25 19:09:30.156136128 +0000 UTC m=+0.596563418 container remove 750410dee27fe8074654c22e23d8564f96c70f73cb8e9b2f946bde7709a0fd1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T19:09:30.655 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T19:09:30.766 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:30.767 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T19:09:30.767 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T19:09:30.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.j 2024-04-25T19:09:31.128 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:30 smithi063 systemd[1]: Stopping Ceph mds.j for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:31.628 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j[68717]: 2024-04-25T19:09:31.303+0000 7f0ace200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:31.628 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:31 smithi063 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j[68717]: 2024-04-25T19:09:31.303+0000 7f0ace200700 -1 mds.j *** got signal Terminated *** 2024-04-25T19:09:33.694 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:33 smithi063 podman[109677]: 2024-04-25 19:09:33.422027632 +0000 UTC m=+2.228702709 container died 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T19:09:33.965 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:33 smithi063 podman[109677]: 2024-04-25 19:09:33.788366957 +0000 UTC m=+2.595042060 container cleanup 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:09:33.965 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:33 smithi063 bash[109677]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j 2024-04-25T19:09:34.377 INFO:journalctl@ceph.mds.j.smithi063.stdout:Apr 25 19:09:33 smithi063 podman[109702]: 2024-04-25 19:09:33.964014443 +0000 UTC m=+0.529635521 container remove 6619c2d134c6d3e5dd4b4013b1fb323d9deab40a72fa94ecf3bf634213bf917f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T19:09:34.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T19:09:34.554 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:34.555 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T19:09:34.555 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T19:09:34.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.b 2024-04-25T19:09:34.836 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:34 smithi070 systemd[1]: Stopping Ceph mds.b for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:35.092 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: 2024-04-25T19:09:34.832+0000 7f96b6a38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:35.092 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:34 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b[63526]: 2024-04-25T19:09:34.832+0000 7f96b6a38700 -1 mds.b *** got signal Terminated *** 2024-04-25T19:09:38.452 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:38 smithi070 podman[73886]: 2024-04-25 19:09:38.166448571 +0000 UTC m=+3.426128629 container died b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T19:09:38.704 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:38 smithi070 podman[73886]: 2024-04-25 19:09:38.533264027 +0000 UTC m=+3.792944083 container cleanup b488c0f6f4b0bd8b434a2cd55dba5faff53ed31fde4c6153c739bc367b386621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=) 2024-04-25T19:09:38.704 INFO:journalctl@ceph.mds.b.smithi070.stdout:Apr 25 19:09:38 smithi070 bash[73886]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-b 2024-04-25T19:09:38.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T19:09:38.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:38.990 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T19:09:38.990 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T19:09:38.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.e 2024-04-25T19:09:39.364 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:39 smithi070 systemd[1]: Stopping Ceph mds.e for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:39.704 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: 2024-04-25T19:09:39.361+0000 7fbf26cd8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:39.704 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:39 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e[62899]: 2024-04-25T19:09:39.361+0000 7fbf26cd8700 -1 mds.e *** got signal Terminated *** 2024-04-25T19:09:42.454 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:42 smithi070 podman[74019]: 2024-04-25 19:09:42.168599703 +0000 UTC m=+2.968027382 container died 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiescer-d69f13d) 2024-04-25T19:09:42.806 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:42 smithi070 podman[74019]: 2024-04-25 19:09:42.52854545 +0000 UTC m=+3.327973136 container cleanup 27594d2271bb35c726a46cb117567461989142fc9587327a01aad1bbee291fc3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-lusov-quiescer-d69f13d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T19:09:42.806 INFO:journalctl@ceph.mds.e.smithi070.stdout:Apr 25 19:09:42 smithi070 bash[74019]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-e 2024-04-25T19:09:42.959 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T19:09:42.992 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:42.992 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T19:09:42.992 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T19:09:42.992 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.h 2024-04-25T19:09:43.454 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:43 smithi070 systemd[1]: Stopping Ceph mds.h for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:43.454 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: 2024-04-25T19:09:43.297+0000 7fb6ea2ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:43.454 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:43 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h[63217]: 2024-04-25T19:09:43.297+0000 7fb6ea2ad700 -1 mds.h *** got signal Terminated *** 2024-04-25T19:09:46.401 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 podman[74140]: 2024-04-25 19:09:46.16649937 +0000 UTC m=+2.959904806 container died 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-25T19:09:46.692 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 podman[74140]: 2024-04-25 19:09:46.506909162 +0000 UTC m=+3.300314587 container cleanup 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-25T19:09:46.692 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 bash[74140]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h 2024-04-25T19:09:46.924 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T19:09:46.944 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 podman[74154]: 2024-04-25 19:09:46.691103167 +0000 UTC m=+0.515700066 container remove 652fc676a66ed1931d64aa6112a9e175342a31b802982d5599fa4727b4790de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T19:09:46.944 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.h.service: Deactivated successfully. 2024-04-25T19:09:46.945 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 systemd[1]: Stopped Ceph mds.h for e90c993c-0331-11ef-bc93-c7b262605968. 2024-04-25T19:09:46.945 INFO:journalctl@ceph.mds.h.smithi070.stdout:Apr 25 19:09:46 smithi070 systemd[1]: ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.h.service: Consumed 21.913s CPU time. 2024-04-25T19:09:46.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:46.956 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T19:09:46.956 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T19:09:46.957 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.k 2024-04-25T19:09:47.204 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:47 smithi070 systemd[1]: Stopping Ceph mds.k for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:47.704 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k[62584]: 2024-04-25T19:09:47.293+0000 7fb2ef73e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:47.704 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:47 smithi070 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k[62584]: 2024-04-25T19:09:47.293+0000 7fb2ef73e700 -1 mds.k *** got signal Terminated *** 2024-04-25T19:09:51.704 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:51 smithi070 podman[74262]: 2024-04-25 19:09:51.299354893 +0000 UTC m=+4.127250739 container died 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T19:09:52.029 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:51 smithi070 podman[74262]: 2024-04-25 19:09:51.785854304 +0000 UTC m=+4.613750151 container cleanup 551e3f29509bb7dd15ed6382b1e00eb6ca1d15d954e7e8811e7845b667e1eef6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T19:09:52.029 INFO:journalctl@ceph.mds.k.smithi070.stdout:Apr 25 19:09:51 smithi070 bash[74262]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-k 2024-04-25T19:09:52.159 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T19:09:52.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:52.191 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T19:09:52.191 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T19:09:52.191 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.c 2024-04-25T19:09:52.486 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:52 smithi114 systemd[1]: Stopping Ceph mds.c for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:52.749 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: 2024-04-25T19:09:52.484+0000 7fdb4ca70700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:52.749 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:52 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c[63410]: 2024-04-25T19:09:52.484+0000 7fdb4ca70700 -1 mds.c *** got signal Terminated *** 2024-04-25T19:09:54.486 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:54 smithi114 podman[73742]: 2024-04-25 19:09:54.173383477 +0000 UTC m=+1.787836537 container died 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-lusov-quiescer-d69f13d, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T19:09:54.804 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:54 smithi114 podman[73742]: 2024-04-25 19:09:54.585305224 +0000 UTC m=+2.199758280 container cleanup 865e95a7b809f4631d954a80eeba42a6b091270f4a625fd83cf26e093baeb60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2) 2024-04-25T19:09:54.804 INFO:journalctl@ceph.mds.c.smithi114.stdout:Apr 25 19:09:54 smithi114 bash[73742]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-c 2024-04-25T19:09:54.912 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T19:09:54.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:54.948 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T19:09:54.948 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T19:09:54.948 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.f 2024-04-25T19:09:55.067 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:55 smithi114 systemd[1]: Stopping Ceph mds.f for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:55.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: 2024-04-25T19:09:55.271+0000 7fc755c56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:55.567 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:55 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f[63102]: 2024-04-25T19:09:55.271+0000 7fc755c56700 -1 mds.f *** got signal Terminated *** 2024-04-25T19:09:58.466 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:58 smithi114 podman[73877]: 2024-04-25 19:09:58.178965513 +0000 UTC m=+3.019779004 container died c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T19:09:58.799 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:58 smithi114 podman[73877]: 2024-04-25 19:09:58.547317034 +0000 UTC m=+3.388130523 container cleanup c2ebf294e58c72cbbf7f0308c49a1ff515b7d46dedfe318a9887bce20c78c068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiescer-d69f13d, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T19:09:58.799 INFO:journalctl@ceph.mds.f.smithi114.stdout:Apr 25 19:09:58 smithi114 bash[73877]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-f 2024-04-25T19:09:58.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T19:09:58.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:09:58.955 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T19:09:58.955 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T19:09:58.955 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.i 2024-04-25T19:09:59.067 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:09:59 smithi114 systemd[1]: Stopping Ceph mds.i for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:09:59.567 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:09:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: 2024-04-25T19:09:59.275+0000 7f88f1a6a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:09:59.567 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:09:59 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i[63717]: 2024-04-25T19:09:59.275+0000 7f88f1a6a700 -1 mds.i *** got signal Terminated *** 2024-04-25T19:10:02.436 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:10:02 smithi114 podman[74000]: 2024-04-25 19:10:02.174336529 +0000 UTC m=+3.009865794 container died 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=wip-lusov-quiescer-d69f13d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-04-25T19:10:02.795 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:10:02 smithi114 podman[74000]: 2024-04-25 19:10:02.526312642 +0000 UTC m=+3.361841910 container cleanup 2a07dcf4c58ec780a8b997bd788351d31e91cdd7d00bae155a7598099ac5c143 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-lusov-quiescer-d69f13d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-25T19:10:02.795 INFO:journalctl@ceph.mds.i.smithi114.stdout:Apr 25 19:10:02 smithi114 bash[74000]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-i 2024-04-25T19:10:02.958 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T19:10:02.990 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:10:02.991 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T19:10:02.991 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T19:10:02.991 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.l 2024-04-25T19:10:03.317 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:03 smithi114 systemd[1]: Stopping Ceph mds.l for e90c993c-0331-11ef-bc93-c7b262605968... 2024-04-25T19:10:03.732 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l[62779]: 2024-04-25T19:10:03.320+0000 7f723c577700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T19:10:03.732 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:03 smithi114 ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l[62779]: 2024-04-25T19:10:03.320+0000 7f723c577700 -1 mds.l *** got signal Terminated *** 2024-04-25T19:10:05.815 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:05 smithi114 podman[74122]: 2024-04-25 19:10:05.429890622 +0000 UTC m=+2.218600952 container died 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiescer-d69f13d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T19:10:06.067 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:05 smithi114 podman[74122]: 2024-04-25 19:10:05.896312936 +0000 UTC m=+2.685023269 container cleanup 530e75fe54271b70391a83441ef14e682d9876568ee9e9833df2f7225dc6f536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:068a558e3af4932cf1c60dd321c0bdf500075eb52ea1bf455e0973ba114fbfd4, name=ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiescer-d69f13d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T19:10:06.067 INFO:journalctl@ceph.mds.l.smithi114.stdout:Apr 25 19:10:05 smithi114 bash[74122]: ceph-e90c993c-0331-11ef-bc93-c7b262605968-mds-l 2024-04-25T19:10:06.257 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e90c993c-0331-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T19:10:06.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T19:10:06.288 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T19:10:06.289 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:10:06.616 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:10:11.928 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:10:12.118 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:10:14.541 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T19:10:14.730 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:10:17.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:10:17.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:10:17.299 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T19:10:17.324 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T19:10:17.327 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi063/crash 2024-04-25T19:10:17.327 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:10:17.424 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:10:17.424 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:10:17.428 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi070/crash 2024-04-25T19:10:17.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:10:17.571 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:10:17.571 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:10:17.573 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi114/crash 2024-04-25T19:10:17.574 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash -- . 2024-04-25T19:10:17.614 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/e90c993c-0331-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T19:10:17.614 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2024-04-25T19:10:17.615 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T19:10:17.616 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T19:10:17.717 INFO:tasks.cephadm:Compressing logs... 2024-04-25T19:10:17.718 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:10:17.722 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:10:17.724 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:10:17.749 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:10:17.749 INFO:teuthology.orchestra.run.smithi070.stderr:: No such file or directory 2024-04-25T19:10:17.749 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:10:17.750 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T19:10:17.751 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T19:10:17.751 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.5% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T19:10:17.756 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:10:17.757 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:10:17.762 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T19:10:17.763 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:10:17.764 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T19:10:17.765 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:10:17.767 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.10.log: 92.5% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T19:10:17.767 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T19:10:17.767 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:10:17.768 INFO:teuthology.orchestra.run.smithi070.stderr: 84.4% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:10:17.768 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T19:10:17.779 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log: 91.0%gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T19:10:17.780 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:10:17.780 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:10:17.780 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log 2024-04-25T19:10:17.781 INFO:teuthology.orchestra.run.smithi063.stderr:: No such file or directory 2024-04-25T19:10:17.781 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T19:10:17.781 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T19:10:17.782 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T19:10:17.783 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T19:10:17.785 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.j.log: 87.5% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:10:17.785 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T19:10:17.787 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T19:10:17.787 INFO:teuthology.orchestra.run.smithi063.stderr: 90.3% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T19:10:17.788 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T19:10:17.789 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 92.1% 94.4% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T19:10:17.789 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:10:17.790 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.x.log: 94.4% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T19:10:17.790 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T19:10:17.791 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T19:10:17.793 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T19:10:17.793 INFO:teuthology.orchestra.run.smithi063.stderr: 94.3% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T19:10:17.794 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T19:10:17.794 INFO:teuthology.orchestra.run.smithi063.stderr: 84.5% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:10:17.795 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T19:10:17.798 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T19:10:17.802 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T19:10:17.803 INFO:teuthology.orchestra.run.smithi063.stderr: 94.3% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T19:10:17.811 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/cephadm.log: 94.3% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T19:10:17.813 INFO:teuthology.orchestra.run.smithi063.stderr: 90.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:10:17.831 INFO:teuthology.orchestra.run.smithi063.stderr: 93.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:10:17.958 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T19:10:18.027 INFO:teuthology.orchestra.run.smithi114.stderr: 92.6% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T19:10:18.196 INFO:teuthology.orchestra.run.smithi063.stderr: 90.2% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T19:10:18.241 INFO:teuthology.orchestra.run.smithi114.stderr: 87.6% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T19:10:18.268 INFO:teuthology.orchestra.run.smithi114.stderr: 94.4% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T19:10:19.155 INFO:teuthology.orchestra.run.smithi114.stderr: 84.3% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T19:10:19.174 INFO:teuthology.orchestra.run.smithi114.stderr: 90.5% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T19:10:19.197 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T19:10:19.255 INFO:teuthology.orchestra.run.smithi063.stderr: 90.1% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T19:10:19.519 INFO:teuthology.orchestra.run.smithi070.stderr: 92.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T19:10:20.801 INFO:teuthology.orchestra.run.smithi114.stderr: 91.6% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T19:10:23.740 INFO:teuthology.orchestra.run.smithi070.stderr: 94.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T19:10:24.746 INFO:teuthology.orchestra.run.smithi070.stderr: 94.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T19:10:27.176 INFO:teuthology.orchestra.run.smithi063.stderr: 93.8% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T19:10:27.289 INFO:teuthology.orchestra.run.smithi063.stderr: 93.7% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T19:10:28.327 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T19:10:28.641 INFO:teuthology.orchestra.run.smithi070.stderr: 94.0% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T19:10:28.941 INFO:teuthology.orchestra.run.smithi063.stderr: 93.8% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T19:10:29.566 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T19:10:29.591 INFO:teuthology.orchestra.run.smithi063.stderr: 93.7% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T19:10:29.595 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-25T19:10:29.596 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m11.842s 2024-04-25T19:10:29.596 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m40.309s 2024-04-25T19:10:29.596 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m3.366s 2024-04-25T19:10:31.932 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T19:10:58.534 INFO:teuthology.orchestra.run.smithi114.stderr: 94.8% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T19:11:04.411 INFO:teuthology.orchestra.run.smithi114.stderr: 94.0% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T19:11:06.852 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T19:11:13.347 INFO:teuthology.orchestra.run.smithi114.stderr: 93.9% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T19:11:28.129 INFO:teuthology.orchestra.run.smithi114.stderr: 94.1% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T19:12:35.196 INFO:teuthology.orchestra.run.smithi070.stderr: 93.2% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T19:12:35.198 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-25T19:12:35.198 INFO:teuthology.orchestra.run.smithi070.stderr:real 2m17.458s 2024-04-25T19:12:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:user 3m12.969s 2024-04-25T19:12:35.199 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m6.106s 2024-04-25T19:13:10.493 INFO:teuthology.orchestra.run.smithi114.stderr: 93.1% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T19:13:18.069 INFO:teuthology.orchestra.run.smithi114.stderr: 93.1% -- replaced with /var/log/ceph/e90c993c-0331-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T19:13:18.071 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-04-25T19:13:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:real 3m0.332s 2024-04-25T19:13:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:user 4m13.631s 2024-04-25T19:13:18.071 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m8.925s 2024-04-25T19:13:18.071 INFO:tasks.cephadm:Archiving logs... 2024-04-25T19:13:18.072 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi063/log 2024-04-25T19:13:18.073 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:13:23.605 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi070/log 2024-04-25T19:13:23.606 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:13:51.067 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi114/log 2024-04-25T19:13:51.068 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:14:31.374 INFO:tasks.cephadm:Removing cluster... 2024-04-25T19:14:31.375 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force 2024-04-25T19:14:31.708 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:14:32.386 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force 2024-04-25T19:14:32.575 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:14:33.054 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e90c993c-0331-11ef-bc93-c7b262605968 --force 2024-04-25T19:14:33.245 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: e90c993c-0331-11ef-bc93-c7b262605968 2024-04-25T19:14:33.771 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T19:14:33.771 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:14:33.832 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:14:33.850 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T19:14:33.865 INFO:tasks.cephadm:Teardown complete 2024-04-25T19:14:33.865 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T19:14:33.879 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T19:14:33.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:14:33.887 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:14:33.892 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:14:34.042 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T19:14:34.056 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T19:14:34.057 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:14:34.065 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:14:34.067 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:14:34.080 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:14:34.081 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +911us[ +917us] +/- 53ms 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -1489us[-1489us] +/- 76ms 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +187us[ +187us] +/- 86ms 2024-04-25T19:14:34.085 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:14:34.098 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:14:34.098 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-04-25T19:14:34.099 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 +900us[ +915us] +/- 53ms 2024-04-25T19:14:34.099 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -1502us[-1502us] +/- 76ms 2024-04-25T19:14:34.099 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +200us[ +200us] +/- 86ms 2024-04-25T19:14:34.099 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:14:34.114 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:14:34.124 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:14:34.125 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-25T19:14:34.125 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +913us[ +928us] +/- 53ms 2024-04-25T19:14:34.125 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -1492us[-1477us] +/- 76ms 2024-04-25T19:14:34.125 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +205us[ +220us] +/- 86ms 2024-04-25T19:14:34.125 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:14:34.126 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T19:14:34.138 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T19:14:34.138 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T19:14:34.151 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:14:34.382 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:14:34.454 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:14:34.554 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:14:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:34.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:14:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:34.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:14:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:34.717 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T19:14:34.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T19:14:34.754 INFO:teuthology.task.internal:Duration was 3074.002989 seconds 2024-04-25T19:14:34.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T19:14:34.785 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T19:14:34.785 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:14:34.795 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:14:34.799 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:14:34.839 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:14:34.853 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:14:34.941 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:14:35.479 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T19:14:35.479 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T19:14:35.479 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:14:35.545 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-25T19:14:35.545 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:14:35.593 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-04-25T19:14:35.593 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:14:35.624 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T19:14:35.624 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:14:35.632 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:14:35.635 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:14:35.721 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T19:14:35.722 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:14:35.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:14:35.731 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:14:36.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T19:14:36.688 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T19:14:36.688 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:14:36.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:14:36.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:14:36.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T19:14:36.928 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:36.936 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:36.939 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:36.963 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-25T19:14:36.966 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-04-25T19:14:37.031 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-04-25T19:14:37.075 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:37.124 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:37.142 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:37.159 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T19:14:37.173 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T19:14:37.174 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi063 2024-04-25T19:14:37.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:14:37.295 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi070 2024-04-25T19:14:37.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:14:37.335 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605/remote/smithi114 2024-04-25T19:14:37.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:14:37.375 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T19:14:37.376 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:14:37.384 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:14:37.386 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:14:37.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T19:14:37.453 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T19:14:37.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T19:14:37.464 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T19:14:37.464 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:14:37.474 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:14:37.477 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:14:37.491 INFO:teuthology.orchestra.run.smithi070.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:14 /home/ubuntu/cephtest 2024-04-25T19:14:37.494 INFO:teuthology.orchestra.run.smithi114.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:14 /home/ubuntu/cephtest 2024-04-25T19:14:37.524 INFO:teuthology.orchestra.run.smithi063.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:14 /home/ubuntu/cephtest 2024-04-25T19:14:37.527 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T19:14:37.540 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T19:14:37.564 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T19:14:37.591 DEBUG:teuthology.nuke:shortname: smithi063 2024-04-25T19:14:37.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:14:37.616 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.580732', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKfD/L/4b3vfTg7Wgk2ts1MsTZ5nXjE5KE7kqkNSEZTL8B94ftKeRrYyw+T2CK6tX/I4vqit23hjc5ULKcbslp4='} 2024-04-25T19:14:37.620 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-25T19:14:37.620 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:14:37.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.581410', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGGyn7n5ycz2BRHDdwHvCt7Z0wYrZku3XnTjAS1GQqPqU814qFVnA+4d7VzeKtfAKImH/g2nP9m2luVnIK6NuuY='} 2024-04-25T19:14:37.649 DEBUG:teuthology.nuke:shortname: smithi114 2024-04-25T19:14:37.650 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:14:37.672 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-25_17:43:55-fs-wip-lusov-quiescer-distro-default-smithi/7673605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 18:13:47.579791', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPhyDAFFEzUC2YHEudlcfE0QYUfX7l9do1WA4GPZdx6d65OSHWRqp+7jZpO37BzXfXZFtZ8GYlYqIfH3ikmSw98='} 2024-04-25T19:14:37.682 INFO:teuthology.orchestra.console.smithi063:Power off 2024-04-25T19:14:37.682 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:14:37.706 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-04-25T19:14:37.706 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:37.730 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-25T19:14:37.731 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:14:37.741 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-04-25T19:14:37.754 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-25T19:14:37.754 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:37.770 INFO:teuthology.orchestra.console.smithi114:Power off 2024-04-25T19:14:37.770 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T19:14:37.780 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-25T19:14:37.792 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-04-25T19:14:37.792 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:37.816 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-25T19:14:41.742 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:41.781 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:41.818 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:41.854 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-04-25T19:14:41.891 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-25T19:14:41.928 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-04-25T19:14:45.860 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:45.891 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:45.929 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:45.970 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-04-25T19:14:46.001 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-25T19:14:46.039 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-04-25T19:14:46.102 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-25T19:14:46.140 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-04-25T19:14:49.974 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T19:14:50.086 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-04-25T19:14:50.187 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-04-25T19:14:50.294 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 3074.0029888153076 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi063 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7797745e6d8b3ac036504dc61491d368615024da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=667113d0dc44406bac414416df5c2b4e status: fail success: false 2024-04-25T19:14:50.294 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:14:50.380 INFO:teuthology.run:FAIL